Light field data acquisition devices, and methods of using and manufacturing same

ABSTRACT

A light field data acquisition device includes optics and a sensor to acquire light field image data of a scene. In at least one embodiment, the sensor is located at a substantially fixed, predetermined location relative to the focal point of the optics. The device determines a first virtual focus depth different from an optical focus depth of the light field image data, and automatically generates and outputs data representative of a first image of the scene at the first virtual focus depth. In response to user input, the device determines a second virtual focus depth and generates data representative of a second image of the scene at the second virtual focus depth.

RELATED APPLICATION

This application claims priority to U.S. Provisional Application Ser.No. 61/120,530, entitled “Light Field Camera and System, and Methods ofUsing and Manufacturing Same”, filed Dec. 8, 2008; and U.S. ProvisionalApplication Ser. No. 61/170,620, entitled “Light Field Camera Image,File and Configuration Data, and Method of Using, Storing andCommunicating Same”, filed Apr. 18, 2009. The contents of theseprovisional applications are incorporated by reference herein, in theirentirety.

INTRODUCTION

In one aspect, the present inventions are directed to, among otherthings, light field data acquisition devices (for example, light fieldcameras) and methods of using and manufacturing such devices. In anotheraspect, the present inventions are directed to characteristics,parameters and configurations of light field data acquisition devices,and methods of using and manufacturing same. Notably, light field dataacquisition devices obtain, acquire, generate, manipulate and/or edit(for example, adjust, select, define and/or redefine the focus and/ordepth of field—after initial acquisition or recording of the image dataand/or information) image data and/or information of, for example, ascene. (See, for example, United States Patent Application Publication2007/0252074, and the provisional application to which it claimspriority, and Ren Ng's PhD dissertation, “Digital Light FieldPhotography”, Stanford University 2006, all of which are incorporatedhere in their entirety by reference; and the block diagram illustrationof a light field camera in FIG. 1).

Optical Notation

A typical characteristic of a light field data acquisition deviceprovides the user the ability to compute images that are sharply focusedover a range of depths, corresponding to a range of virtual image planesabout the physical plane where the light field sensor was positioned.With reference to FIG. 2A, this range of sharp focusing corresponds tothe range of (virtual) image plane depths a distance of ε about thephysical light field sensor plane. In FIG. 2A:

-   -   Lens plane may be characterized as the principal plane of the        lens; it may be advantageous to employ thin-lens simplifications        of lenses in the illustrative diagrams, although the concepts        apply to any lens configuration and/or system;    -   Far-focus plane may be characterized as the virtual plane        optically conjugate to the furthest objects in the world that        can be brought into a predetermined focus, for example, sharply        into focus) using post image data acquisition focusing        techniques of the light field;    -   Focal plane may be characterized as the plane in which rays        emanating from optical infinity are brought into sharpest focus        by the optics.    -   Light field sensor plane may be characterized as the plane in        the data acquisition device where the principal plane of the        microlens array in the light field sensor (e.g. microlens array        and image sensor assembly) is physically located;    -   Close-focus plane may be characterized as the virtual plane        optically conjugate to the closest objects in the world that can        be brought sharply into focus through software focusing of the        light field;    -   v is equal to the distance between the lens plane and the light        field sensor plane; and    -   ε₁ and ε₂ are equal to the maximum distances from the light        field sensor plane that can be focused sharply after        exposure—that is, after acquisition of image data.

Notably, in FIG. 2A, the “world” or everything outside of the lightfield data acquisition device is drawn to the left of the lens plane,and the device internals are illustrated to the right of the lens plane.Moreover, FIG. 2A is not drawn to scale; indeed, ε₁ and ε₂ are oftensmaller than v (for example, ε₁<0.01*v and ε₂<0.01*v).

As intimated herein, although the present inventions are often describedin the context of a light field capture system or device, which acquireor obtain refocusable data or information and/or processes or methods ofacquiring, generating, manipulating and/or editing such refocusableimage data or information (i.e., post image data acquisition focusingtechniques), it should be clear that the present inventions areapplicable to other systems, devices, processes and/or methods ofacquiring, generating, manipulating and/or editing refocusable imagedata or information. In this regard, refocusable image data orinformation are image data or information, no matter how acquired orobtained, that may be focused and/or re-focused after acquisition orrecording of the data or information. For example, in one embodiment,refocusable image data or information is/are light field data orinformation acquired or obtained, for example, via a light field dataacquisition device.

SUMMARY OF CERTAIN ASPECTS OF THE INVENTIONS

There are many inventions described and illustrated herein. The presentinventions are neither limited to any single aspect nor embodimentthereof, nor to any combinations and/or permutations of such aspectsand/or embodiments. Moreover, each of the aspects of the presentinventions, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose permutations and combinations will not be discussed separatelyherein.

Briefly, a light field data acquisition device according to certainaspects and/or embodiments of the present inventions includes deviceoptics (e.g. one or more lenses of any kind or type), sensors to obtainand/or acquire the light field data or information, and circuitry toprocess the light field data or information. For example, in oneembodiment of the present inventions, a light field data acquisitiondevice includes optics, having one or more, or all of the followingelements/features:

-   -   a. Main lens such as the image-forming lens on a conventional        camera; and/or    -   b. Image sensor fixed at “light field hyperfocal” distance;        and/or    -   c. Microlens having a predetermined resolution, for example, a        resolution determined using one or more of the techniques of        using a human face or other selected, desired or predetermined        object or subject as the field of view size of a predetermined        refocusable distance (for example, a closest refocusable        distance). The microlens array may correspond to, for example,        an array of convex, image-forming lenses, and it may be        positioned between the main lens and the image sensor at a        separation from the sensor surface of approximately the focal        length of each microlens in the microlens array.

In such a light field data acquisition device, the sensor and microlensassembly may be referred to herein as a “light field sensor.” (See, forexample, FIG. 2B).

In other aspects and/or embodiments, the data acquisition deviceaccording to certain aspects and/or embodiments of the presentinventions includes circuitry (for example, properly programmedprocessing circuitry) to generate, manipulate and/or edit (for example,adjust, select, define and/or redefine the focus and/or depth offield—after initial acquisition or recording of the image data and/orinformation) image data and/or information of, for example, a scene.

Notably, the term “circuit” may mean, among other things, a singlecomponent (for example, electrical/electronic) or a multiplicity ofcomponents (whether in integrated circuit form, discrete form orotherwise), which are active and/or passive, and which are coupledtogether to provide or perform a desired function or operation. The term“circuitry” may mean, among other things, a circuit (whether integrated,discrete or otherwise), a group of such circuits, one or more processors(digital signal processors (DSPs)), one or more state machines, one ormore processors implementing software, one or more Application-SpecificIntegrated Circuits (ASICs), one or more programmable gate arrays (PGAs)(for example, field-programmable gate arrays (FPGAs), and/or acombination of one or more circuits (whether integrated, discrete orotherwise), one or more state machines, one or more processors, and/orone or more processors implementing software, one or more ASICs, one ormore PGAs. Moreover, the term “optics” means a system comprising aplurality of components used to affect the propagation of light,including but not limited to lens elements, windows, apertures andmirrors.

The light field data acquisition device according to certain aspectsand/or embodiments of the present inventions may include “automaticfocusing” after acquisition of the image data or information (i.e.,after the shot has been taken). The “automatic focusing” may employvarious techniques including but not limited to:

-   -   a. Scene analysis auto-focus;    -   b. Scene analysis depth of field selection; and/or    -   c. Scene analysis tilt/shift focal plane selection.        As such, in this embodiment, processing circuitry in the light        field data acquisition device, may determine, calculate, assess,        generate, derive and/or estimate one or more predetermined,        appropriate and/or selected focus(es) (i.e., one or more        (virtual) image plane depths or focal planes) after acquisition        of the image data or information (i.e., after the shot has been        taken), based on, for example, an analysis of one or more        characteristics of the scene corresponding to the image data or        information.

In addition thereto, or in lieu thereof, the light field dataacquisition device according to certain aspects and/or embodiments ofthe present inventions may include circuitry and/or implement methods tofacilitate interaction for “user-guided” and/or “user-defined” one ormore focus selections/determinations and/or depth of fieldselections/determinations. For example, in the context of focusselection, the light field data acquisition device may include circuitryand/or implement techniques to:

-   -   a. Touch-screen click/selection to designate, identify and/or        select one or more objects/subjects to focus on;    -   b. “Push/pull” focus—that is, drag the focus nearer or closer;        and/or    -   c. Modify, edit and/or adjust a depth of field by specifying one        or more ranges of depths to, for example, bring one or more        objects or subjects into a predetermined focus.

In addition thereto, the light field data acquisition device and systemaccording to certain aspects and/or embodiments of the presentinventions may include circuitry and/or perform methods to implementlive-viewing—before capture, sampling and/or acquisition of the lightfield image data which is employed to generate the final image. Forexample, the light field data acquisition device and system mayimplement:

-   -   a. Extended depth of field (EDOF) by, for example, selective        sampling of the acquired/observed array of pixel values on the        sensor in a light field data acquisition device; and/or    -   b. Focusing techniques on one or more objects or subjects of        interest.        In one exemplary embodiment of the live viewing, the light field        data acquisition device according to certain aspects of the        present inventions may be employed as follows:    -   1. The user or operator may point the light field data        acquisition device at, for example, a target and set a        predetermined and/or desired zoom position, for example, to        frame an object or a subject.    -   2. The user or operator may instruct or enable the data        acquisition device to obtain or acquire image data or        information by, for example, “clicking” the shutter button. In        response, the light field data acquisition device acquires or        obtains image data of a picture/scene without a delay that is        typically associated with cameras having physical auto focus        optics and/or circuitry. The operation of the light field data        acquisition device may be designed such that all other        operations, some of which may typically occur during the delay        of physical auto focus (e.g. exposure metering, auto-exposure        calculations and/or exposure settings), occur prior to the click        of the shutter button so as to minimize and/or eliminate all        delay between “clicking” the shutter button and the light field        data acquisition device acquiring the image data of information        of a picture/scene.    -   3. The user, operator and/or light field data acquisition device        may store, edit and/or analyze the acquired or recorded light        field data or information; for example, such data or information        may be edited to adjust, select, define and/or redefine the        focus and/or depth of field using the system (as properly        programmed). Indeed, a user may edit or manipulate the acquired        or recorded light field data or information to focus (manually        or automatically) on one or more subjects or objects of        interest, which may be visually presented to the user via a        display (for example, an LCD) that may be disposed on the light        field data acquisition device. The user may, before and/or after        editing (if any), store the light field data or information (or        a representation thereof) in internal or external memory (for        example, an external memory storage that is coupled to the data        acquisition device).    -   4. The user or operator may instruct the light field data        acquisition device to, for example, focus or refocus on one or        more different subjects/objects by, for example, selecting,        clicking and/or pointing on a “touch screen” disposed on the        data acquisition device or system. Notably, the user may also        instruct the light field data acquisition device or system to        edit other characteristics of the acquired image—for example,        adjust the depth of field (for example, increase or decrease the        depth of field).

Importantly, the present inventions are neither limited to any singleaspect nor embodiment, nor to any combinations and/or permutations ofsuch aspects and/or embodiments. Moreover, each of the aspects of thepresent inventions, and/or embodiments thereof, may be employed alone orin combination with one or more of the other aspects and/or embodimentsthereof. For the sake of brevity, many of those permutations andcombinations will not be discussed and/or illustrated separately herein.

Notably, it should be clear that certain of the present inventions ofgenerating, manipulating and/or editing light field image data orinformation are applicable to light field data acquisition devices andsystems physically configured according to one or more of the exemplaryembodiments of the present inventions and/or data acquisition devicesand systems not physically configured according to one or more of theexemplary embodiments of the present inventions. In this regard, thetechniques of generating, manipulating and/or editing light field imagedata or information may be implemented on data acquisition devices andsystems according to one or more of the exemplary embodiments describedand/or illustrated herein as well as any data acquisition device and/orsystem that provides the capability to edit or manipulate image data orinformation, for example, focus an image to a range of focal depthsafter “exposure” or acquisition of image data.

In a first principle aspect, certain of the present inventions aredirected to a light field imaging device for acquiring light field imagedata of a scene, the device of this aspect includes optics, wherein theoptics includes an optical path and a focal point, wherein the focalpoint is associated with a focal length of the optics, and light fieldsensor to acquire light field image data in response to a first userinput and located at a substantially fixed, predetermined locationrelative to the focal point of the optics, wherein the predeterminedlocation is substantially independent of the scene. The optical depth offield of the optics with respect to the light field sensor extends to adepth that is closer than optical infinity. The device also includesprocessing circuitry, coupled the user interface, to: (a) determine afirst virtual focus depth of the light field image data, wherein thefirst virtual focus depth is different from the optical focus depth ofthe light field image data, (b) automatically generate data which isrepresentative of a first image of the scene using the light field imagedata, wherein the first image includes a focus which corresponds to thefirst virtual focus depth, (c) output the data which is representativeof the first image, and, after outputting the data which isrepresentative of the first image and in response to the second userinput, (d) determine a second virtual focus depth of the light fieldimage data using data which is representative of the second user input,wherein the second user input is indicative of the second virtual focusdepth, and (e) generate data which is representative of a second imageof the scene which includes a focus that corresponds to the secondvirtual focus depth.

Notably, in response to outputting the first image, the user interfacereceives the second user input which is indicative of the second virtualfocus depth. In this regard, the user interface may include a display to(i) receive the data which is representative of the first image and (ii)output the first image. Moreover, the light field imaging device mayinclude memory, coupled to the processing circuitry, to store the datawhich is representative of the second image.

In one embodiment, the optics is configurable to include a plurality ofdifferent focal lengths having associated focal points. In thisembodiment, the substantially fixed, predetermined location of the lightfield sensor, relative to the focal point of the optics, changes inaccordance with the focal length of the optics.

In another embodiment, the optics may include a zoom lens system (forexample, providing a continuous or non-continuous zoom) having aplurality of zoom positions. Here, the device may include a mechanicalsystem (an active or a passive system), coupled to the light fieldsensor, to maintain the light field sensor unit at the same fixed,predetermined location relative to the focal point of the optics for theplurality of the zoom positions. The mechanical system may maintain thelight field sensor unit at the predetermined location relative to thefocal point of the optics for the plurality of the zoom positions of thezoom lens system. Indeed, the mechanical system may maintain the lightfield sensor unit at the predetermined location relative to the focalpoint of the optics for a plurality of focuses of the optics. Notably,the light field sensor, in these embodiments, may maintain thepredetermined location relative to the focal point of the optics for theplurality of the zoom positions of the zoom lens system.

The light field imaging device of this aspect of the inventions mayfurther include (i) a spatial adjustment unit, coupled to the lightfield sensor and/or the optics, to responsively move the light fieldsensor unit, and (ii) control circuitry, coupled to the spatialadjustment unit, to control the spatial adjustment unit to maintain thelight field sensor unit at the predetermined location relative to thefocal point of the optics. The device may also include memory, coupledto the control circuitry, to store data which is representative of aplurality of locations of the light field sensor corresponding to aplurality of associated zoom positions. Indeed, the memory may include adatabase or lookup table which correlates a plurality of locations ofthe light field sensor according to a plurality of associated zoompositions. Here, the control circuitry may access the database or lookuptable having data which is representative of the location based on thezoom position of the zoom lens system and generates control signal whichare applied to the spatial adjustment unit to responsively move thelight field sensor unit.

In one embodiment, the processing circuitry of the light field imagingdevice computes output images having an array of output pixels, andwherein the light field sensor includes (i) a microlens array having aplurality of microlenses and (ii) a sensor having an array of sensorpixels, and wherein the predetermined distance is between(0.7*N_(usable)*W_(MLA)*F#_(optics)*m)/(W_(output)) and(3.0*N_(usable)*W_(MLA)*F#_(optics)*m)/(W_(output)),

where:

-   -   N_(usable) is equal to the number of pixels containing usable        directional information across a microlens disk image that is        formed, through each microlens, on the sensor, or are located        under a microlens,    -   W_(MLA) is representative of a number of microlenses across a        width of the microlens array,    -   W_(output) is representative of a number of pixels across a        width of the array of output pixels,    -   m is equal to a distance between the centers of two neighboring        microlenses in the microlens array, and    -   F#_(optics) is equal to an F-number of the optics.

In another embodiment, the light field sensor includes (i) a microlensarray having a plurality of microlenses and (ii) a sensor having anarray of pixels, and wherein the predetermined distance is greater than(m*F#_(optics)),

where:

-   -   m is equal to the distance between the centers of two        neighboring microlenses in the microlens array, and    -   F#_(optics) is equal to an F-number of the optics.

In yet another embodiment, the light field sensor includes (i) amicrolens array having a plurality of microlenses and (ii) a sensorhaving an array of pixels, and wherein the predetermined distance isless than (N_(usable)*m*F#_(optics)),

where:

-   -   N_(usable) is equal to the number of pixels containing usable        directional information across a microlens disk image that is        formed, through each microlens, on the sensor, or are located        under a microlens,    -   m is equal to the distance between the centers of two        neighboring microlenses in the microlens array, and    -   F#_(optics) is equal to an F-number of the optics.

In another principal aspect, the present inventions are directed to alight field imaging device for acquiring light field image data of asubject in a scene, wherein the subject is in a focal plane, the deviceof this aspect of the inventions comprises: (a) optics, wherein theoptics includes an optical path, an optical depth of field and anautofocus mechanism, wherein the autofocus selects between a pluralityof focuses including a first focus which is related to a focal plane ofthe subject, (b) a light field sensor, located (i) in the optical pathof the optics to acquire light field image data and (ii) at apredetermined location, relative to the optics, during acquisition ofthe light field data, wherein the predetermined location is at asubstantially fixed separation relative to the focal plane correspondingto the subject, and wherein the focal plane of the subject is outside ofthe optical depth of field of the optics with respect to the light fieldsensor, and (c) processing circuitry to generate an output image datawhich is representative of an image including the subject in the scene,the processing circuitry to: (i) determine a virtual focus depth of thelight field image data, wherein the virtual focus depth is differentfrom the optical focus depth of the light field image data, and (ii)automatically generate data which is representative of the image of thescene using the light field image data and data which is representativeof the virtual focus depth, wherein the image includes a focus whichcorresponds to the virtual focus depth.

Notably, the light field imaging device may also include memory, coupledto the processing circuitry, to store the data which is representativeof the image of the scene.

In one embodiment, the light field imaging device further includes auser interface to receive a plurality of user inputs, including firstand second user inputs, wherein, in response to the first user input,the light field sensor acquires the light field image data of the scene,and wherein the processing circuitry outputs the data which isrepresentative of the image of the scene, and, after outputting the datawhich is representative of the image of the scene and in response to thesecond user input, (i) determines a final virtual focus depth of thelight field image data using data which is representative of the seconduser input, wherein the second user input is indicative of the finalvirtual focus depth, and (ii) generates data which is representative ofthe final image of the scene using the light field image data whereinthe final image includes a focus which corresponds to the final virtualfocus depth.

The light field imaging device may also include control circuitry andwherein the autofocus mechanism includes an autofocus image sensor toacquire focus data, the control circuitry determines focus metrics usingthe focus data, and the optics includes a second optics to provide atleast a portion of the light in the optical path onto the autofocusimage sensor.

In yet another principal aspect, the present inventions are directed toa light field imaging device for acquiring light field image data of asubject in a scene, wherein the subject is in a focal plane, the devicecomprises: (a) optics, wherein the optics includes an optical path, anoptical depth of field and an autofocus mechanism, wherein the autofocusselects between a plurality of focuses including a first focus which isrelated to a focal plane of the subject, (b) a light field sensor,located (i) in the optical path of the optics to acquire light fieldimage data and (ii) at a predetermined separation, relative to theoptics, during acquisition of the light field data, wherein the focalplane of the subject is outside of the optical depth of field of theoptics with respect to the light field sensor, (c) control circuitry toadjust the separation between the light field sensor and the opticsbetween a plurality of discrete, fixed-focus separations, wherein eachdiscrete separation provides a refocusing range of the light field imagedata, and (d) processing circuitry to generate an output image datawhich is representative of an image including the subject in the scene,the processing circuitry to: (i) determine a virtual focus depth of thelight field image data, wherein the virtual focus depth is differentfrom the optical focus depth of the light field image data, and (ii)automatically generate data which is representative of the image of thescene using the light field image data and data which is representativeof the virtual focus depth, wherein the image includes a focus whichcorresponds to the virtual focus depth.

In one embodiment of this aspect of the invention, the discrete,fixed-focus separations overlap in refocusing range of the light fieldimage data.

In another embodiment, the control circuitry selects a discrete,fixed-focus separation based on which of the discrete, fixed-focusseparations provides a range of refocusing which is the furthest towardsoptical infinity and the in-focus plane of the subject is within therange of refocusing of the selected discrete, fixed-focus separation.Here, the autofocus mechanism includes an autofocus image sensor toacquire focus data, wherein the control circuitry determines focusmetrics using the focus data, and the optics includes a second optics todivert at least a portion of the light in the optical path onto theautofocus image sensor.

Again, the light field imaging device may further include a userinterface to receive a plurality of user inputs, including first andsecond user inputs, wherein, in response to the first user input, thelight field sensor acquires the light field image data of the scene, andwherein the processing circuitry outputs the data which isrepresentative of the image of the scene, and, after outputting the datawhich is representative of the image of the scene and in response to thesecond user input, (i) determines a final virtual focus depth of thelight field image data using data which is representative of the seconduser input, wherein the second user input is indicative of the finalvirtual focus depth, and (ii) generates data which is representative ofthe final image of the scene using the light field image data whereinthe final image includes a focus which corresponds to the final virtualfocus depth.

Moreover, the light field imaging device may also include controlcircuitry and wherein the autofocus mechanism includes an autofocusimage sensor to acquire focus data, the control circuitry determinesfocus metrics using the focus data, and the optics includes a secondoptics to provide at least a portion of the light in the optical pathonto the autofocus image sensor.

In yet another principal aspect, the present inventions are directed toa light field imaging device for acquiring light field image data of ascene, wherein the light field image device includes a maximum outputimage resolution, the device comprising: (a) optics, wherein the opticsincludes an optical path and a focal point, (b) a light field sensor,located (i) in the optical path of the optics to acquire light fieldimage data and (ii) at a substantially fixed location relative to thefocal point of the optics, wherein the substantially fixed location issubstantially independent of the scene and creates an optical depth offield of the optics with respect to the light field sensor that extendsto a depth that is closer than optical infinity. The light field imagingdevice of this aspect of the inventions also includes processingcircuitry to generate and output image data which is representative ofan output image of the scene, the processing circuitry to: (i) determinea virtual focus depth of the light field image data, wherein the virtualfocus depth is different from the optical focus depth of the light fieldimage data, and (ii) automatically generate data which is representativeof the output image of the scene using the light field image data anddata which is representative of the virtual focus depth, wherein theoutput image includes a focus which corresponds to the virtual focusdepth. In this aspect of the inventions, the light field sensor, whichis located at the substantially fixed location relative to the focalpoint of the optics, acquires light field image data which correspondsto an output image of the scene which includes a virtual focus ofoptical infinity and a resolution in lines per picture height (LPH) thatis at least 0.5 times the maximum output image resolution of the lightfield imaging device.

The maximum output image resolution of the device may be a number ofrows of pixels in an image output by the device. The maximum outputimage resolution of the device may be the maximum resolution in LPH ofany image output by the device virtually refocused to any subject depth.

Notably, the light field sensor, which is located at the substantiallyfixed location relative to the focal point of the optics, may acquirelight field image data which corresponds to an output image of the scenewhich includes a focus of optical infinity and a resolution in lines perpicture height (LPH) that is less than 0.8 times the maximum outputimage resolution of the light field imaging device

Again, there are many inventions, and aspects and embodiments of theinventions, described in this Summary and/or described and/orillustrated herein. This Summary is not exhaustive of the scope, aspectsand/or embodiments of the present inventions. Indeed, this Summary maynot be reflective of or correlate to the inventions protected by theclaims in this or in continuation/divisional applications hereof.

Moreover, this Summary is not intended to be limiting of the inventionsor the claims (whether the currently presented claims or claims of adivisional/continuation application (if any)) and should not beinterpreted in that manner. While certain embodiments have beendescribed and/or outlined in this Summary, it should be understood thatthe present inventions are not limited to such embodiments, descriptionand/or outline, nor are the claims limited in such a manner (whichshould also not be interpreted as being limited by this Summary).

Indeed, many other aspects, inventions and embodiments, which may bedifferent from and/or similar to, the aspects, inventions andembodiments presented in this Summary, will be apparent from thedescription, illustrations and claims, which follow. In addition,although various features, attributes and advantages have been describedin this Summary and/or are apparent in light thereof, it should beunderstood that such features, attributes and advantages are notrequired whether in one, some or all of the embodiments of the presentinventions and, indeed, need not be present in any of the embodiments ofthe present inventions.

BRIEF DESCRIPTION OF THE DRAWINGS

In the course of the detailed description to follow, reference will bemade to the attached drawings. These drawings show different aspects ofthe present inventions and, where appropriate, reference numeralsillustrating like structures, components, materials and/or elements indifferent figures are labeled similarly. It is understood that variouscombinations of the structures, components, materials and/or elements,other than those specifically shown, are contemplated and are within thescope of the present inventions.

Moreover, there are many inventions described and illustrated herein.The present inventions are neither limited to any single aspect norembodiment thereof, nor to any combinations and/or permutations of suchaspects and/or embodiments. Moreover, each of the aspects of the presentinventions, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose permutations and combinations will not be discussed and/orillustrated separately herein.

FIG. 1 is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions;

FIG. 2A is a block diagram representation of certain opticalcharacteristics of an exemplary light field data acquisition deviceincluding certain focus planes such as a far-focus plane, a physicallight field sensor plane, and the close-focus plane;

FIG. 2B is a block diagram representation of an exemplary light fieldsensor including, among other things, a microlens array and imagingsensor, which may be separated by (or substantially separated by) thefocal length of the microlens array, according to at least certainaspects of certain embodiments of the present inventions and/or whichmay implement certain aspects of certain embodiments of the presentinventions;

FIG. 2C is a block diagram representation of the light field sensorplane, which may be disposed at the principal plane of the microlensarray, according to at least certain aspects of certain embodiments ofthe present inventions and/or which may implement certain aspects ofcertain embodiments of the present inventions;

FIG. 3A is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions;

FIG. 3B is a block diagram representation of an exemplary light fielddata acquisition device including, among other things, processingcircuitry integrated therein, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions;

FIGS. 3C and 3D are block diagram representations of exemplary lightfield data acquisition device wherein the processing circuitry is notintegrated therein, according to at least certain aspects of certainembodiments of the present inventions and/or which may implement certainaspects of certain embodiments of the present inventions;

FIG. 3E is a block diagram representation of exemplary light field dataacquisition device, according to at least certain aspects of certainembodiments of the present inventions and/or which may implement certainaspects of certain embodiments of the present inventions, wherein thelight field data acquisition device couples to external systems/devices(for example, external storage, video display, recording device and/ordata storage);

FIG. 3F is a block diagram representation of an exemplary systemincluding light field data acquisition device and post-processingcircuitry, according to at least certain aspects of certain embodimentsof the present inventions and/or which may implement certain aspects ofcertain embodiments of the present inventions;

FIG. 4 is a block diagram representation of certain opticalcharacteristics of an exemplary light field data acquisition device,according to at least certain aspects of certain embodiments of thepresent inventions and/or which may implement certain aspects of certainembodiments of the present inventions, including certain focus planessuch as an infinity focal plane, a light field hyperfocal plane (focalplane of the sensor), and a near image focus plane;

FIG. 5A is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the exemplary light field data acquisition device of thisembodiment includes a mechanical system that mechanically couples to theoptics and/or the microlens array and/or the photo sensor to provide ormaintain the value of ε (the distances between the in-focus focal planeand the light field sensor plane) constant or substantially constantacross a predetermined zoom range of the exemplary light field dataacquisition device, and the lens plane of optics (via, for example, afixture) maintains or provides the focal point of the optics (i.e.position of infinity focus) in a fixed location or position relative tothe light field sensor plane as the zoom position changes (i.e. thefocal length F changes) in response to, for example, a user input;

FIG. 5B is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the mechanical system includes one or more fixtures thatmechanically couple to one or more elements of the optics to provide ormaintain the value of ε (the distance between the in-focus focal planeand the light field sensor plane) constant or substantially constantacross a predetermined zoom range of the exemplary light field dataacquisition device, and the lens plane of optics (via, for example, afixture) maintains or provides the focal point of the optics (i.e.position of infinity focus) in a fixed location or position relative tothe light field sensor plane as the zoom position changes (i.e. thefocal length F changes) in response to, for example, a user input;

FIG. 5C is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the offset between the focal plane and light field sensor may bekept constant or substantially constant for each acquisition or captureby mechanically adjusting the location of the optics relative to thelight field sensor after a change in the zoom configuration;

FIG. 5D is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the offset between the focal plane and light field sensor may bekept constant or substantially constant for each acquisition or captureby utilizing a main lens system that maintains a constant or nearconstant focal plane for a plurality of zoom positions;

FIG. 5E is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the optics includes a zoom lens configuration or apparatus (forexample, a conventional type zoom lens configuration or apparatus) andthe exemplary light field data acquisition device of this embodimentincludes control circuitry to derive, calculate and/or estimate apredetermined or required focus based on a desired, predetermined and/ornew zoom position of the zoom lens configuration and, in response,adjust (for example, automatically) the focus of the optics tocorrespond to the predetermined or required focus;

FIG. 5F is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the optics includes a zoom lens configuration or apparatus (forexample, a conventional type zoom lens configuration or apparatus) andthe exemplary light field data acquisition device of this embodimentincludes control circuitry and memory that stores a predetermineddatabase which maps or correlates zoom positions of the zoom lensconfiguration of the optics to the appropriate or predetermined focussettings;

FIGS. 5G and 5H are block diagram representations of exemplary lightfield data acquisition devices, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the mechanical system includes one or more fixtures thatmechanically couple to one or more elements or portions of the optics tocontrol, adjust, move, provide and/or maintain a predetermined positionof the optics relative to the sensor and microlens array, for example,to control, adjust, move, provide and/or maintain the focal point of theoptics (i.e. position of infinity focus) in a predetermined location orposition relative to the light field sensor plane to provide apredetermined, discrete and/or fixed focus of the light field dataacquisition device (in response to, for example, a user input);

FIG. 5I is a block diagram representation of an exemplary light fielddata acquisition device, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the mechanical system includes one or more fixtures thatmechanically couple to the sensor and microlens array to control,adjust, move, provide and/or maintain a predetermined position of thesensor and microlens array relative to the optics to, for example,control, adjust, move, provide and/or maintain the focal point of theoptics (i.e. position of infinity focus) in a predetermined location orposition relative to the light field sensor plane to provide apredetermined, discrete and/or fixed focus of the light field dataacquisition device (in response to, for example, a user input);

FIG. 5J-5L are block diagram representations of exemplary light fielddata acquisition devices, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the mechanical system includes one or more fixtures thatmechanically couple to one or more elements of the optics, and thesensor and microlens array to control, adjust, move, provide and/ormaintain a predetermined relative position of the optics and/or sensorand microlens array to, for example, control, adjust, move, provideand/or maintain the focal point of the optics (i.e. position of infinityfocus) in a predetermined location or position relative to the lightfield sensor plane to provide a predetermined, discrete and/or fixedfocus of the light field data acquisition device (in response to, forexample, a user input);

FIGS. 5M and 5N are block diagram representations of exemplary lightfield data acquisition devices, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions,wherein the light field data acquisition devices include an auto focussystem (for example, a conventional type auto focus architecture) andwherein, in certain embodiments, the control circuitry may coordinatethe operation of the auto focus system in conjunction with the positionof the optics and/or sensor (and microlens array) to provide apredetermined, selected and/or desired focus of the light field dataacquisition device;

FIG. 6 illustrates an exemplary embodiment of a portion of a light fielddata acquisition device that utilizes a zoom lens architecture orassembly, in three exemplary zoom configurations corresponding to threedifferent focal lengths, according to at least certain aspects ofcertain embodiments of the present inventions, wherein in each of theillustrated configurations, as in every zoom configuration, the lightfield sensor is positioned at substantially the same distance ε from thelens plane passing through the focal point of the optics correspondingto that zoom configuration;

FIG. 7 illustrates an exemplary embodiment of a portion of a light fielddata acquisition device having an optics section (or portion thereof)which is controllable, adjustable and/or moveable, relative to, forexample, the sensor, between a plurality of discrete positions which mayprovide a plurality of corresponding discrete focus “positions” orcharacteristics, according to at least certain aspects of certainembodiments of the present inventions, wherein each exemplary focusconfiguration corresponds to different discrete focus “positions” orcharacteristics;

FIG. 8 illustrates an exemplary embodiment of a portion of a light fielddata acquisition device having a sensor which is controllable,adjustable and/or moveable, relative to, for example, the optics (orportion or elements thereof), between a plurality of discrete positionswhich may provide a plurality of corresponding discrete focus“positions” or characteristics, according to at least certain aspects ofcertain embodiments of the present inventions, wherein each exemplaryfocus configuration corresponds to different discrete focus “positions”or characteristics;

FIG. 9A is a block diagram representation of certain opticalcharacteristics of an exemplary light field data acquisition device,according to at least certain aspects of certain embodiments of thepresent inventions and/or which may implement certain aspects of certainembodiments of the present inventions, including the relationshipbetween certain focus planes as the position or location of the sensormoves between a plurality of discrete positions which may provide aplurality of corresponding discrete focus “positions” orcharacteristics;

FIG. 9B is a block diagram representation of certain opticalcharacteristics of an exemplary light field data acquisition device,according to at least certain aspects of certain embodiments of thepresent inventions and/or which may implement certain aspects of certainembodiments of the present inventions, including the relationshipbetween certain focus planes as the position or location of the sensormoves between positions corresponding to an “infinity mode” and a “macromode” which may provide a plurality of corresponding discrete focus“positions” or characteristics;

FIG. 10A-10C are block diagram representations of certain opticalcharacteristics of exemplary light field data acquisition devices,according to at least certain aspects of certain embodiments of thepresent inventions and/or which may implement certain aspects of certainembodiments of the present inventions, including the relationshipbetween certain focus planes as the position or location of the sensor(and microlens array) move between a plurality of course-graineddiscrete positions which correspond to overlapping or non-overlappingrefocusing ranges wherein the separation of the discrete focuspositions, relative to neighboring position(s), provides (i) one or moreoverlapping refocusing position(s) and/or (ii) one or morenon-overlapping refocusing position(s);

FIG. 11A is a block diagram representation of the relative separation ofthe lens plane and the light field sensor in an exemplary light fielddata acquisition device including an auto-focus system, according to atleast certain aspects of certain embodiments of the present inventionsand/or which may implement certain aspects of certain embodiments of thepresent inventions, wherein the device starts with a first separation(for example, the separation for the previous capture or acquisition)and using autofocus, adjusts the separation such that there may be asecond separation where the subject may be in optical focus on the lightfield sensor, and prior to acquisition, the separation may be adjustedby a predetermined offset to set a third separation;

FIG. 11B is a block diagram representation of an exemplary light fielddata acquisition device containing a second image sensor used forautomatic focusing, according to at least certain aspects of certainembodiments of the present inventions and/or which may implement certainaspects of certain embodiments of the present inventions;

FIG. 11C is a block diagram representation of the relative separation ofthe lens plane and the light field sensor in an exemplary light fielddata acquisition device including an auto-focus system, according to atleast certain aspects of certain embodiments of the present inventionsand/or which may implement certain aspects of certain embodiments of thepresent inventions, wherein the device starts with a first separation(for example, the separation for the previous capture or acquisition)and using autofocus, adjusts the separation such that there may be asecond separation where the subject may be in optical focus on the lightfield sensor, and prior to acquisition, the separation may be adjusted adistance equal or nearly equal to ε to set a third separation;

FIG. 12 is an illustrative representation of a view display (forexample, view finder and/or live-view LCD or the like display) of theuser interface of an exemplary light field data acquisition device,according to at least certain aspects of certain embodiments of thepresent inventions and/or which may implement certain aspects of certainembodiments of the present inventions;

FIG. 13 illustrates the nature of the light captured in relation to themicrolens F-number and the main lens F-number, wherein when the F-numberof the main lens is higher than the F-number of a microlens of themicrolens array, the “disk images” that appear on and are captured bythe sensor are smaller, which may reduce the directional resolution ofthe acquired, sampled and/or captured light field;

FIGS. 14A, 14B and 14C illustrate exemplary disk image projections and arelationship between the F-number of the main lens and the F-number of amicrolens of the microlens array on the acquired, sampled and/orcaptured light field by the sensor; wherein, in contrast with the“full-sized” disk images of the microlens array as projected on theassociated sensor array (FIG. 14A), when the F-number of the main lensis higher than the F-number of a microlens of the microlens array, thelight field projects smaller disk images on the associated sensor arrayand the unshaded areas of the disk images (FIG. 14B), which correspondto non-illuminated sensor pixels that do not acquire, sample and/orcapture light field data and, as such, do not contribute to the imageand may be characterized as “wasted” or “unused” pixels or sensors, andcorrespondingly, when the F-number of the main lens is smaller than theF-number of a microlens of the microlens array, the light field projectslarger disks on the associated sensor array (FIG. 14C);

FIG. 15 illustrates, in a block diagram manner, optics and light fieldsensor of an exemplary light field data acquisition device including amechanical aperture that may change the size of the aperture during zoomin order for the optics to maintain a constant F-number as the zoomconfiguration changes, according to at least certain aspects of certainembodiments of the present inventions and/or which may implement certainaspects of certain embodiments of the present inventions;

FIG. 16 illustrates, in a block diagram manner, certain aspects of aplurality of exemplary operations, functions and/or embodiments ofafter-the-shot focusing or refocusing;

FIGS. 17A and 17B show images corresponding to a region of light fielddata containing an image of an eye, wherein, in contrast in FIG. 17A,the plane of focus for the eye and the light field sensor plane are notaligned while in FIG. 17B the plane of focus for the eye and the lightfield sensor plane are aligned;

FIG. 18 is a block diagram representation showing how the number ofdiscrete focus positions may be determined based on the main lens, aclose focus distance, and ε, according to at least certain aspects ofcertain embodiments of the present inventions and/or which may implementcertain aspects of certain embodiments of the present inventions;

FIG. 19A is an illustration of a readout mode on a light field sensorthat combines the center four pixels in each light field disk image andgenerates an RGB or YUV image from the data;

FIG. 19B is an illustration of a readout mode on a light field sensorthat reads the center four pixels in each light field disk image andgenerates a smaller Bayer pattern mosaic image from the data;

FIG. 20 is an illustration indicating how a pixel from or under eachmicrolens may be selected to create or form an image of a scene with alarge depth of field for example, for “live” view; notably, the image onthe left is a zoomed and magnified section of a light field image withpixel near center of each projected microlens disk highlighted; theblack box near the center of each disk is a representation of anexemplary technique of selecting a pixel near or at the center and thelines and image on the right conceptually demonstrate how the individualpixels from the disk centers (in this exemplary embodiment) may beassembled into a final image that may be employed, for example, inconnection with “live” view; and

FIG. 21 is an image of an exemplary resolution chart used to measurelines per picture height (LPH).

Again, there are many inventions described and illustrated herein. Thepresent inventions are neither limited to any single aspect norembodiment thereof, nor to any combinations and/or permutations of suchaspects and/or embodiments. Each of the aspects of the presentinventions, and/or embodiments thereof, may be employed alone or incombination with one or more of the other aspects of the presentinventions and/or embodiments thereof. For the sake of brevity, many ofthose combinations and permutations are not discussed separately herein.

DETAILED DESCRIPTION

There are many inventions described and illustrated herein, as well asmany aspects and embodiments of those inventions. In one aspect, thepresent inventions are directed to the physical optical design of thelight field acquisition device or system (hereinafter “device” and“system” are collectively “device” unless indicated otherwise). Inanother aspect, the present inventions are directed to methods ofoperating a light field data acquisition device including userinteraction therewith and “live-view” or real-time processing of imagesprior to collection, acquisition and/or sampling of the final lightfield image data or information by the light-field acquisition device.In yet another aspect, the present inventions are directed to techniquesand/or methods of “after-the-shot automatic” focusing.

Exemplary Use of Light Field Data Acquisition Device

In one exemplary embodiment, the light field data acquisition device 10,according certain aspects of the present inventions may be, among otherthings or ways, employed as follows:

-   -   1. The user or operator may point the light field data        acquisition device at, for example, a target and set a        predetermined and/or desired “zoom position”, for example, to        frame an object or a subject. Briefly, a zoom position refers to        a configuration of optics allowing a variable or configurable        focal length (for example, a zoom lens), such that a particular        focal length is selected. Herein, changing and/or varying the        zoom position may be used interchangeably with changing and/or        varying the focal length of optics allowing a variable or        configurable focal length.    -   2. The user or operator may instruct or enable the data        acquisition device to obtain or acquire image data or        information via the user interface (for example, by “clicking”        or engaging the shutter button). In response, the light field        data acquisition device acquires or obtains image data of a        picture/scene which may be without a delay that is typically        associated with conventional cameras (for example, conventional        cameras having physical auto focus optics and/or circuitry). The        operation of the light field data acquisition device may be        designed such that all other operations, some of which may        typically occur during the delay of physical auto focus (e.g.        exposure metering, auto-exposure calculations and/or exposure        settings), occur prior to the click of the shutter button so as        to reduce, minimize and/or eliminate all delay between        “clicking” or engaging the shutter button and the light field        data acquisition device acquiring the image data or information        of a picture/scene.    -   3. The user, operator and/or light field data acquisition device        may store, edit and/or analyze the acquired or recorded light        field data or information; for example, such data or information        may be edited to adjust, select, define and/or redefine the        focus and/or depth of field using the system (as properly        programmed). Indeed, a user may edit or manipulate the acquired        or recorded light field data or information to focus (manually        or automatically) on one or more subjects or objects of        interest, which may be visually presented to the user via a        display (for example, an LCD) that may be disposed on the light        field data acquisition device. The user may, before and/or after        editing (if any), store the light field data or information (or        a representation thereof) in internal or external memory (for        example, an external memory storage that is coupled to the data        acquisition device).    -   4. The user or operator may instruct the light field data        acquisition device to, for example, focus or refocus on one or        more different subjects/objects by, for example, selecting,        clicking and/or pointing on a “touch screen” disposed on the        data acquisition device or system. Notably, the user may also        instruct the light field data acquisition device or system to        edit other characteristics of the acquired image—for example,        adjust the depth of field (for example, increase or decrease the        depth of field).        Notably, the operation described above may be employed in        conjunction with any of the light field data acquisition device        embodiments described and/or illustrated herein. For the sake of        brevity, the discussion of the operation will not be repeated in        conjunction with each light field data acquisition device        embodiments described and/or illustrated herein.        Light Field Data Acquisition Device Optics

Briefly, with reference to FIGS. 3A-3E, light field data acquisitiondevice 10 may include optics 12 (including, for example, a main lens),light field sensor 14 including microlens array 15 and sensor 16 (forexample, a photo sensor). The microlens array 15 is incorporated intothe optical path to facilitate acquisition, capture, sampling of,recording and/or obtaining light field data via sensor 16. Notably, thelight field data acquisition discussions set forth in United StatesPatent Application Publication 2007/0252074, the provisionalapplications to which it claims priority (namely, U.S. ProvisionalPatent Application Ser. Nos. 60/615,179 and 60/647,492), and Ren Ng'sPhD dissertation, “Digital Light Field Photography”) are incorporatedherein by reference.

The light field data acquisition device 10 may also include controlcircuitry to manage or control (automatically or in response to userinputs) the acquisition, sampling, capture, recording and/or obtainingof light field data. The light field data acquisition device 10 maystore the light field data (for example, output by sensor 16) inexternal data storage and/or in on-system data storage. All permutationand combinations of data storage formats of the light field data and/ora representation thereof are intended to fall within the scope of thepresent inventions.

Notably, light field data acquisition device 10 of the presentinventions may be a stand-alone acquisition system/device (see, FIGS.3A, 3C, 3D and 3E) or may be integrated with processing circuitry (see,FIGS. 3B and 3F). That is, light field data acquisition device 10 may beintegrated (or substantially integrated) with processing circuitry whichmay be employed to generate, manipulate and/or edit (for example,adjust, select, define and/or redefine the focus and/or depth offield—after initial acquisition or recording of the light field data)image data and/or information of, for example, a scene; and, in otherexemplary embodiments, light field data acquisition device 10 isseparate from processing circuitry which generates, manipulates and/oredits, for example, the focus and/or depth of field, after initialacquisition or recording of the light field data.

Notably, a light field data acquisition device, during capture and/oracquisition, may have a light field sensor located such that the“optical depth of field” with respect to the light field sensor does notinclude the location of a subject. Briefly, the “optical depth of field”may be characterized as depth of field the device would have if used asa conventional imaging device containing a conventional imaging sensor.

As noted above, in one aspect, the present inventions are directed tothe physical and/or optical design of light field data acquisitiondevice 10. In one embodiment of this aspect of the present inventions,light field data acquisition device 10 includes optics, microlens arrayand sensor configuration that provides a fixed focus in connection withthe acquisition of light field image data. In this regard, withreference to FIGS. 3A and 4, the physical positions or locations ofelements of optics 12 (which, as mentioned above, may include one ormore lens, windows, apertures and mirrors) and/or light field sensor 14,comprised of microlens array 15 and sensor 16, are substantially fixed,in a relative manner, to, for example, provide a fixed focus duringacquisition of light field image data by light field data acquisitiondevice 10. The relative positions of optics 12 and/or light field sensor14 may be selected to provide a range of refocusing after image dataacquisition or a post-acquisition range of focusing or re-focusing ofthe data or information (via, for example, post-processing circuitry) toextend from a first predetermined distance from an element of optics 12(for example, a predetermined distance from lens plane 18 (for example,infinity—which may be characterized as optical infinity or essentiallyoptical infinity given the observer's perceptive ability in targetoutput images) which, for illustrative purposes, is illustrated as frommain lens 20 to a second predetermined distance from lens plane 18 (forexample, as close as possible to the lens plane yet still provides aselected, suitable or predetermined resolution). Notably, herein theterm ε may be used to indicate the distance from the light field sensorplane 24 to a virtual plane of focus (for example, the focal plane ofthe optics or the virtual plane of focus corresponding to an object orsubject). Generally, ε may be used without subscript in the context of asingle distance. In the context of a plurality of light field sensorplanes and/or virtual planes of focus, ε may be used with subscript (forexample, ε₁ and ε₂) to denote the different distances.

Notably, with reference to FIG. 2C, the location of the light fieldsensor plane 24 may be considered the same as the principal plane of theelements in the microlens array 15. Herein, the location of light fieldsensor plane 24 may be referred to as the location and/or placement ofthe light field sensor 14 (for example, when describing the locationand/or placement relative to other components and/or modules in thelight field data acquisition device (for example, optics 12)).

In one embodiment, the position or location of light field sensor 14, ina relative manner, may be referred to as the “light field hyperfocal”focus position wherein a first predetermined distance may be opticalinfinity and a second predetermined distance may be as close to the lensplane which provides a selected, suitable or predetermined resolution.With reference to FIG. 4, light from an object at optical infinityconverges at in-focus focal plane 22, and light field sensor 14 may bepositioned at light field sensor plane 24, which may be located orpositioned a distance of ε₁ (which may be, for example, measured inmillimeters or micrometers) away to facilitate or enable refocusing frominfinity to a predetermined close distance. The predetermined closedistance may be characterized as the plane conjugate to the image plane(or near image focal plane 26) which is at a distance F+ε₁+ε₂ from thelens plane. Briefly, for a light field sensor including a microlensarray and an image sensor, ε₁ and/or ε₂ may be characterized and/ordetermined as described below.

Define the following:

-   -   Let m equal the distance between the centers of two neighboring        microlenses in the microlens array (in millimeters).    -   Let N equal the number of sensor pixels across a microlens disk        image that is formed through the microlens on the sensor, or are        located under a microlens.    -   Let F#_(optics) equal the F-number of the main lens.    -   Let F#_(MLA) equal the F-number of the microlens array.    -   Let N_(usable) equal the number of pixels containing usable        directional information across a microlens disk image that        appears or are located under a microlens. In the case where the        F-numbers of the main lens and microlens array are equal or        nearly equal, N_(usable) may equal N. In cases where the        F-numbers are not equal or near equal, N_(usable) may be less        than N. In some embodiments, N_(usable) may be calculated by:        N _(usable) =N*min(F# _(MLA) /F# _(optics),2.0−F# _(MLA) /F#        _(optics))

where min is a function that takes two arguments and returns the smallerof the two values. The maximum amount of blur that is introduced on thesensor from a point that comes into focus a distance of ε away may begenerally described by the following:b=ε/(F# _(optics))Thus, expressing ε as a function of b gives the maximum distance thatthe sensor can be placed from the desired image plane to permit theobject to be brought into a predetermined focus using a light fieldprocessing technique:ε=b(F# _(optics))To produce a refocused image, the maximum blur size b that may beovercome by post acquisition refocusing of the light field image data(via processing circuitry) may be characterized approximately asfollows:b=KN _(usable) m,such thatε=KN _(usable) m(F# _(optics))where K is a constant factor that may vary depending on various factors(for example, the manufacturing process and/or the desired outputresolution), and will be discussed later herein.

Notably, in some embodiments, the effective and/or desired value for K(and thus ε) may differ depending on the direction in which the focus isshifted. As an example, in a light field data device containingdirectionally sensitive pixels on the imaging sensor, it may be possibleto generate refocused images with higher resolution for objects in theforeground than objects in the background, given the same opticalmisfocus. In these embodiments, ε₁ may be different than ε₂, but isgenerally within a small multiple (for example, ε₁<ε₂<2*ε₁).

Notably, if a lower-resolution image is computed, then the maximum imageblur that may be overcome by post-acquisition refocusing increases, andthus ε may be greater. A related phenomenon is that refocusing at depthscorresponding to sensor distances greater than ε results in images thathave incrementally lower-resolution, but that may be acceptable,suitable and/or desired for, for example, certain applications. Ingeneral, the performance of the light field data acquisition device isaffected by other parameters, including output image resolution, desiredimage sharpness, refocusing range, lens focal length and aperture,sensor size, microlens diameter, and/or macro mode magnification. Thechoices for these parameters may impact the distance ε (i.e., the rangeof (virtual) image plane depths about the physical light field sensorplane).

In this embodiment, light field sensor 14 may be positioned relative tothe infinity focal plane of optics 12 (for example, a distance of F+ε)using a plurality of mechanical and electromechanical configurations.For example, with reference to FIGS. 4 and 5A, one exemplary embodimentemploys mechanical system 28 (which may be passive or active), which isphysically coupled to one or more elements of optics 12 and/or lightfield sensor 14 to provide or maintain the value of ε (the distancebetween in-focus focal plane 22 and light field sensor plane 24)constant or substantially constant across a predetermined zoom range oflight field data acquisition device 10. In one embodiment, withreference to FIG. 5B, fixture 30 physically couples to one or moreelements of optics 12 and, in combination with spatial adjustment unit32 (for example, stepper motors, microelectromechanical device,cooperating levers or arms, voicecoil motors, ultrasonic motor (USM),arc-form drive (AFD), and/or micromotor (MM)) maintains or provides thefocal point of optics 12 (i.e. position of infinity focus) in a fixedlocation or position relative to light field sensor plane 24 as the zoomposition changes (i.e. the focal length F changes) in response to, forexample, a user input. In this way, light field sensor 14 (and lightfield sensor plane 24) is located or positioned a constant (orsubstantially constant) distance ε beyond or further from the focalpoint of optics 12 (for example, the focal point of lens 20). Forexample, FIG. 6 illustrates an exemplary embodiment of a portion of alight field data acquisition device that utilizes a zoom lens, in threeexemplary zoom configurations corresponding to three different focallengths. In each of the illustrated configurations, as in every zoomconfiguration, light field sensor 14 is positioned at substantially thesame distance ε from lens plane 18 passing through the focal point ofoptics 12 corresponding to that zoom configuration.

Briefly, the focal point of the optics may be characterized as thelocation where parallel light rays converge after passing through theoptics. Generally, optics determine two focal points, with one on eachside of the optics. Herein, the term focal point refers to the focalpoint inside of the device, determined by rays originating in the world.Further, in certain optical configurations that may contain opticalaberrations such that there may be no well-defined point of convergence,the focal point may be characterized as the point that has the lowestroot mean squared (RMS) error, where the error may be measured as thedistance from the point to lines determined by light rays originatingfrom optical infinity (i.e. parallel light rays in the world).

In one embodiment, with reference to FIG. 5C, the light field dataacquisition device may adjust the location of optics 12 relative tolight field sensor 14 each time the zoom configuration changes, whichmay place the light field sensor a predetermined distance from focalplane 22 at the time of acquisition. In this regard, the light fielddata acquisition device may be initially in one zoom position, withlight field sensor 14 a predetermined distance, ε, from focal plane 22.The zoom configuration may change (for example, as a result of a userinput), which may place focal plane 22 at a different separation fromlight field sensor 14. In this embodiment, before acquisition of thelight field data (for example, immediately or substantially immediatelyafter changing the zoom configuration), the location of light fieldimage sensor 14 relative to focal plane 22 may be adjusted to placelight field sensor 14 at a predetermined distance ε relative to focalplane 22. (See, “Second Zoom Configuration—Final”) Notably, thenecessary adjustment and/or light field sensor position may be stored ina database or look-up table using the zoom position as a key and basedthereon, the light field data acquisition device may implement theappropriate adjustments to provide a predetermined distance ε relativeto focal plane 22. (See, for example, the control circuitry and memoryof FIG. 5F).

In another exemplary embodiment of a passive mechanical system wherein εvaries with the zoom position, the physical focus changes in synchronywith a change in zoom such that the light field sensor plane ismaintained or kept a predetermined distance from the lens plane. Here,the zoom lens may be designed to change the physical focus as the zoomchanges. In addition thereto, or in lieu thereof, a second lens isconfigured to change in synchrony with a change in zoom (for example,with one or more other elements of the optics (for example, the zoomlens)) such that the light field sensor 14 maintains a predeterminedrelative distance from the lens plane. For example, with reference toFIG. 5D, focal plane 22 maintains a constant position, relative to lightfield sensor 14, over a plurality of zoom configurations. Notably, manyzoom lens systems are designed such that the focal plane is constant ornearly constant across the range of zoom positions.

In another exemplary embodiment of a passive mechanical system, the zoomlens of the zoom system includes the property whereby the focal plane ofthe lens is maintained at approximately a fixed plane in space as thelens is “zoomed,” i.e. the focal length of the lens is changed. In thisembodiment, the light field sensor is maintained at a constant or nearconstant ε separation from this focal plane via a passive mechanicalspacer, bracket or other mechanical structure that offsets the lightfield sensor depth from the lens' focal plane.

Notably, a number of exemplary embodiments have been described, withboth passive and/or active components and methods, to illustrate thegeneral principle of maintaining a desired or predetermined separationbetween the focal plane of the lens and the light field sensor. Anymethod now known or later invented for maintaining this desiredseparation is intended to fall within the scope of this aspect of thepresent inventions.

The light field data acquisition device may also include a conventionalzoom lens configuration, including separate control of zoom and focus.In this regard, with reference to FIGS. 4 and 5E, in one embodiment,such a light field data acquisition device 10 may independently controlthe zoom lens and focus via control circuitry 34. In the context ofcontrol circuitry 34, when the lens of optics 12 is located or moved toa desired, predetermined and/or new zoom position, the focus of optics12 (zoom lens) may be adjusted or changed such that light field sensorplane 24 is located or placed an appropriate or predetermined distancefrom the lens plane 18. The control circuitry 34 may determine, derive,calculate and/or estimate a predetermined or required focus based on adesired, predetermined and/or new zoom position of the zoom lensconfiguration of optics 12 and, in response, adjust (for example,automatically) the focus of the lens of optics 12 to correspond to thepredetermined or required focus.

In another embodiment, control circuitry 34 (for example, a processor,state machine, ASIC, PGA (for example, field-programmable gate array(FPGA)) may be configured, capable and/or programmed to control the lenszoom based on the user's interaction with the zoom input interface (forexample, knobs/dials of user interface 36 disposed on or connected withlight field data acquisition device 10) to automatically set the lensfocus or focus of optics 12 as the user selects/input the desired,predetermined and/or new zoom. In one embodiment, the correlationbetween zoom position and focus may be determined empirically oranalytically based on the mathematical relationship. In anotherembodiment, the correlation data may be stored in a memory (for example,memory that is integrated in circuitry (for example, control circuitry34) or that is discrete) such that a predetermined database maps orcorrelates zoom positions of the zoom lens configuration of optics 12 tothe appropriate or predetermined focus settings. (See, for example,FIGS. 5E and 5F).

Notably, in some embodiments, the value of ε may vary with the zoomposition of the optics 12. Briefly, as the focal length becomes greater,the depth of field in the “world” generally becomes shallower. In someembodiments, it may be desirable to use larger values of ε for longerfocal lengths to provide a larger range of refocusing relative tosmaller values of ε. However, as noted herein, using a larger value forε may result in some portions of the refocusing range (for example, nearand far extremes) that have lower resolution and/or appear less sharp.

In sum, in one aspect, the present inventions are directed to a singlefixed-focus lens configuration with the sensor placed a predetermineddistance from a predetermined image plane (such as the plane passingthrough the lens' focal point) to provide a predetermined and/or desiredrefocusing performance of light field data acquisition device 10.

Optics and/or Sensor Moves Over Coarse-Grained Focus Positions: Inanother embodiment of the present inventions, optics 12 and/or lightfield sensor 14 are/is adjustable or moveable, in a relative manner,between a plurality of discrete, fixed positions which may provide aplurality of corresponding discrete, fixed-focus “positions” orcharacteristics. For example, with reference to FIGS. 3A and 7, in afirst predetermined focus (for example, the focus configurationcorresponding to Exemplary Focus Configuration 1), the elements ofoptics 12 (for example, one or more lenses) are located (relative to theplurality of elements of optics 12 and/or light field sensor 14) in afirst position. In a second predetermined focus (for example, the focusconfiguration corresponding to Exemplary Focus Configuration 2), theelements of optics 12 (for example, one or more lenses) are located(relative to the plurality of elements of optics 12 and/or light fieldsensor 14) in a second position, such that lens plane 18 is moved adistance Δx₁ relative to the location of lens plane 18 corresponding toExemplary Focus Configuration 1. Similarly, in a third predeterminedfocus (the focus configuration corresponding to Exemplary FocusConfiguration 3), the elements of optics 12 (for example, one or morelenses) are located (relative to the plurality of elements of optics 12and/or light field sensor 14) in a third position, such that lens plane18 is moved (i) a distance Δx₂ relative to the location of lens plane 18corresponding to Exemplary Focus Configuration 1 and (ii) a distance Δx₃relative to the location of lens plane 18 corresponding to ExemplaryFocus Configuration 2. Each of the discrete locations provides apredetermined focus configuration of the light field data acquisitiondevice.

Notably, one or more of the elements of optics 12 may be positioned (ina relative manner) using any circuitry, mechanisms and/or techniques nowknown or later developed, including the circuitry, mechanisms and/ortechniques described herein (see, for example, FIG. 5B). In theseembodiments, however, the elements of optics 12 are moved or positionedbetween a plurality of discrete locations or positions to provide aplurality of corresponding fixed focus characteristics. Moreover,although the exemplary illustration reflects the relative motion of theentire optics 12, certain of the elements of optics 12 (for example, oneor more lenses) may be spatially fixed (for example, relative to lightfield sensor 14 and other portions/elements of optics 12). (See, forexample, FIGS. 5G and 5H).

In another embodiment of the present inventions, light field dataacquisition device 10 includes light field sensor 14 which is adjustableor moveable, relative to optics 12, between a plurality of fixedpositions to provide a plurality of corresponding discrete, fixed-focus“positions” or characteristics. For example, with reference to FIGS. 3Aand 8, in a first predetermined focus (for example, the focusconfiguration corresponding to Exemplary Focus Configuration 1), theelements of optics 12 (for example, one or more lenses) are spatiallyfixed and light field sensor 14 is located (relative to the plurality ofelements of optics 12) in a first position. In a second predeterminedfocus (for example, the focus configuration corresponding to ExemplaryFocus Configuration 2), light field sensor 14 is located in a secondposition or location relative to optics 12, such that light field sensorplane 24 is moved a distance Δx₁ relative to the location of light fieldsensor plane 24 corresponding to Exemplary Focus Configuration 1.Similarly, in a third predetermined focus (the focus configurationcorresponding to Exemplary Focus Configuration 3), light field sensor 14is located (relative to optics 12) in a third position, such that lightfield sensor plane 24 is moved (i) a distance Δx₂ relative to thelocation of lens plane 18 corresponding to Exemplary Focus Configuration1 and (ii) a distance Δx₃ relative to the location of light field sensorplane 24 corresponding to Exemplary Focus Configuration 2. Each of thediscrete locations provides a predetermined focus configuration of thelight field data acquisition device.

Notably, light field sensor 14 may be positioned (in a relative manner)using any of circuitry, mechanisms and/or techniques now known or laterdeveloped, including the circuitry, mechanisms and/or techniquesdescribed herein (see, for example, FIGS. 5I-5L). In these embodiments,however, light field sensor 14 is moved or positioned between aplurality of discrete locations or positions to provide a plurality ofcorresponding fixed focus characteristics. Moreover, although theexemplary illustration reflects the movement of light field sensor 14relative to optics 12, optics 12 (or one or more elements thereof, forexample, one or more lenses) may also move (for example, otherportions/elements of optics 12). (See, for example, FIGS. 5K and 5L).

In another embodiment, with reference to FIGS. 3A and 9A, when imagingan object at or near optical infinity, the image is in a predeterminedfocus (for example, in-focus) on image plane v_(infinity) and lightfield sensor 14 is placed at image plane v_(infinity)+ε_(infinity)(i.e., corresponding to light field sensor plane 24 a). When imaging acloser object, however, the image is in focus at image plane v_(object)and light field sensor 14 is placed at image plane v_(object)+ε_(object)(i.e., corresponding to light field sensor plane 24 b).

With continued reference to FIGS. 3A and 9A, in one exemplaryembodiment, light field data acquisition device 10 may include aplurality of fixed-focus positions including a first position for“refocus to infinity” mode and a second position for “macro” mode. Theuser may select or define the focus configuration of light field dataacquisition device 10 by selection of the mode, for example, via a userinterface 36. (See, for example, FIGS. 3B, 3D and 7). In thisembodiment, with reference to FIG. 9B, when light field data acquisitiondevice 10 is configured (i) in a first mode (infinity mode), light fielddata acquisition device 10 includes a focus characteristic that extendsto infinity (or substantially infinity), and (ii) in a second mode(macro mode), the refocusing range of light field data acquisitiondevice 10 does not extend to infinity but the closest refocusabledistance is closer to optics 12 of light field data acquisition device10 relative to the configuration corresponding to the first mode.Notably, the refocusing ranges covered in the refocusing modes may beadjacent, overlapping (for example, as in the diagram), ornon-overlapping.

This embodiment may be implemented via a moveable optics section and/ormoveable light field sensor 14. In this regard, the optics 12 and/or thelight field sensor 14 may be moveable or adjustable, in a relativemanner, between a plurality of discrete positions which provide aplurality of corresponding discrete, fixed-focus “positions” orcharacteristics corresponding to the plurality of modes. Although theillustrated embodiment intimates that the sensor is moveable between aplurality of positions or locations, such movement is relative to theoptics section (for example, the lens plane 18) and, as such, theseembodiments may be implemented via a moveable optics section and a fixedor stationary sensor (and microlens array). (See, for example, FIGS. 5B,5G and 5H).

Notably, it may be advantageous when the light field data acquisitiondevice is in a macro mode, to provide a widened main lens aperture suchthat the effective or apparent f-number of the aperture of main lens ofthe optics “matches” the f-number of the aperture of the microlens ofthe microlens array. A general approximation of the effective orapparent f-number of the main lens aperture is as follows:Microlens F#=f/T _(micro)→Main lens F#=f/T _(main)

-   -   where: M is the magnification, and T_(main)=T_(micro)/(1+M).

For example, in one embodiment, the magnification M=1, each microlens isf/4, and the main lens is opened up to f/2. In this embodiment, thelight field disk images are full-sized.

In another embodiment, the light field data acquisition device 10includes a plurality of coarse-grained fixed-focus (with overlapping ornon-overlapping regions of refocusability) positions. With reference toFIGS. 5A and 10A, in one exemplary embodiment, the separation of thefixed-focus positions provides a plurality of refocusable overlapping inrange (see, for example, v₁ and v₂, and v₂ and v₃). That is, in thisexemplary embodiment, when light field sensor 14 is located in a firstposition (sensor position 1), the refocusing range of light field dataacquisition device 10 corresponds to +ε₁₋₂ and −ε₁₋₁ relative to lightfield sensor plane 24 a. Similarly, when light field sensor 14 islocated in a second position (sensor position 2), the refocusing rangeof light field data acquisition device 10 corresponds to +ε₂₋₂ and −ε₂₋₁relative to light field sensor plane 24 b, wherein the refocusing rangeof light field data acquisition device 10 when light field sensor 14 islocated in the first position (sensor position 1) overlaps with therefocusing range of light field data acquisition device 10 when lightfield sensor 16 is located in the second position (sensor position 2).

Further, when light field sensor 14 is located in a third position(sensor position 3), the refocusing range of light field dataacquisition device 10 corresponds to +ε₃₋₂ and −ε₃₋₁ relative to lightfield sensor plane 24 c and the refocusing range of light field dataacquisition device 10 when light field sensor 14 is located in the thirdposition (sensor position 3) overlaps with the refocusing range of lightfield data acquisition device 10 when light field sensor 14 is locatedin the second position (second position 2). In sum, in this illustrativeexemplary embodiment, light field data acquisition device 10 includesthree discrete, fixed-focus overlapping positions.

With continued reference to FIGS. 3A and 10A, in one embodiment, whencapturing or acquiring image data or information using the light fielddata acquisition device of FIG. 10A, light field sensor 14 may bepositioned at v₁ (sensor position 1), v₂ (sensor position 2), or v₃(sensor position 3) thereby enabling light field data acquisition device10 to provide or produce a refocusable image corresponding to anycontinuous position from v₁−ε₁₋₁ to v₃+ε₃₋₂ (which is the aggregate ofthe refocusable range when the photo sensor is located in sensorpositions 1, 2 and 3). Where light field data acquisition device 10includes an auto focus system (see, for example, FIGS. 5M and 5N), thecontrol circuitry 34 (which may control the auto focus system) of lightfield data acquisition device 10 may select between one of a pluralityof discrete, fixed focus positions, rather than, for example, afine-grained continuum of focus positions. Notably, increasing theoverlap between the refocusable ranges of the discrete positions of thedata acquisition device of these embodiments, may reduce, lessen, and/orlighten any accuracy requirements or constraints for a physicalauto-focus system implemented in light field data acquisition device 10.

It should be noted that this embodiment may be implemented via amoveable optics section and/or moveable light field sensor 14. In thisregard, the optics and/or the light field sensor may be moveable oradjustable, in a relative manner, between a plurality of discrete, fixedpositions which provide a plurality of corresponding discrete,fixed-focus “positions” or characteristics corresponding to theplurality of overlapping course-grained fixed-focus positions. Althoughthe illustrated embodiment intimates that the light field sensor 14 ismoveable between a plurality of positions or locations (see, forexample, FIGS. 5I-5L), such movement is relative to the optics section(for example, lens plane 18) and, as such, these embodiments may beimplemented via a moveable optics section and a fixed or stationarysensor (and microlens array). (See, for example, FIGS. 5B, 5G and 5H).

In another embodiment of the present inventions, the light field dataacquisition device includes a plurality of coarse-grained fixed-focuswherein one or more (or all) of the fixed-focus positions arenon-overlapping with respect to the fixed-focus positions correspondingto the neighboring discrete position of the optics and/or sensor (andmicrolens array). For example, with reference to FIGS. 10B and 10C,light field data acquisition device 10 includes one or more discrete,fixed-focus positions that provide non-overlapping refocusable ranges(see, for example, v₂ and v₃) with respect to neighboring positions(see, sensor position 2 and sensor position 3). In this exemplaryembodiment, when light field sensor 14 is located in a first position(sensor position 1), the refocusing range of light field dataacquisition device 10 corresponds to +ε₁₋₂ and −ε₁₋₁ relative to lightfield sensor plane 24 a, which overlaps with the refocusing range oflight field data acquisition device 10 when light field sensor 14 islocated in a second position (sensor position 2). However, in thisembodiment, when light field sensor 14 is located in a third position(sensor position 3), the refocusing range of light field dataacquisition device 10 corresponds to +ε₃₋₂ and −ε₃₋₁ relative to lightfield sensor plane 24 c, which does not overlap with the refocusingrange of light field data acquisition device 10 when light field sensor14 is located in a second position (sensor position 2).

Notably, with reference to FIG. 10C, when light field sensor 14 islocated in a fourth position (sensor position 4), the refocusing rangeof light field data acquisition device 10 corresponds to +ε₄₋₂ and −ε₄₋₁relative to light field sensor plane 24 d, which overlaps with therefocusing range of light field data acquisition device 10 when lightfield sensor 14 is located in a third position (sensor position 3).

As with the implementation of the overlapping position (for example,discussed immediately above and illustrated in FIG. 10A), theembodiments of FIGS. 10B and 10C may be implemented via a moveableoptics section and/or moveable light field sensor 14. In this regard,the optics and/or the light field sensor 14 may be moveable oradjustable, in a relative manner, between a plurality of fixed positionswhich provide a plurality of corresponding discrete, fixed-focus“positions” or characteristics corresponding to the plurality ofnon-overlapping course-grained fixed-focus positions. Although theillustrated embodiment describes that the light field sensor 14 ismoveable between a plurality of positions or locations (see, forexample, FIGS. 5I-5L), such movement is relative to the optics section(for example, the lens plane 18) and, as such, these embodiments may beimplemented via a moveable optics section and a fixed or stationarysensor (and microlens array). (See, for example, FIGS. 5B, 5G and 5H).

Where light field data acquisition device 10 includes a zoom lens (see,for example, FIGS. 5E and 5F) and more than one fixed focus positionsoverlap (see, for example, FIGS. 10A and 10B), the number of overlappingfixed focus positions that may be necessary to allow any object from agiven minimum close distance to infinity to be brought into apredetermined focus may increase as the focal length becomes longer, andmay decrease as the focal length becomes shorter. In general, there maybe some focal length threshold in the zoom range of the optics at whicha single fixed focus position may be all that is required, and at thisthreshold and at any shorter focal lengths, light field data acquisitiondevice 10 may switch to behaving as a device with a single fixed focusposition, as described above.

Notably, for a given light field data acquisition device 10, in general,the number of fixed focus positions that are employed to span or providea given refocusable range may be a function of the focal length of themain lens of optics 12.

In another embodiment, the light field data acquisition device,according to at least certain aspects of certain embodiments of thepresent inventions and/or which may implement certain aspects of certainembodiments of the present inventions, includes a physical auto-focus,and optics/circuitry to implement such auto-focus, in those embodimentswhere the light field data acquisition device includes multiplefixed-focus positions (for example, three positions in the exemplaryillustrated embodiment of FIGS. 10A and 10B). In another embodiment, thelight field data acquisition device includes a physical auto-focus, andoptics/circuitry to implement such auto-focus, in those embodimentswhere the light field data acquisition device includes independent zoomand focus controls in which a single fixed-focus position (e.g.refocusing from optical infinity down to a close distance) does notprovide or cover a sufficiently large focus range. The light field dataacquisition devices of these embodiments may employ any physicalauto-focus now known or later developed; all of which, when incorporatedinto the light field data acquisition devices of these embodiments, areintended to fall within the scope of the present inventions.

Notably, a light field data acquisition device with a discrete, fixed orcoarse focus mechanism may be able to reduce, minimize and/or eliminatethe time spent searching for the correct focus prior to acquisition orcapture. In some embodiments, the operation of the light field dataacquisition device may be designed such that all other operations, someof which may typically occur during the delay of physical auto focus(e.g. exposure metering, auto-exposure calculations and/or exposuresettings), occur prior to the click of the shutter button so as tominimize and/or eliminate all delay between “clicking” the shutterbutton and the light field data acquisition device acquiring the imagedata of information of a picture/scene. In one embodiment, the lightfield data acquisition device may periodically acquire or capture“analysis” images in order to determine, calculate and/or perform tasks(e.g. exposure metering, auto-exposure calculations and/or exposuresettings) that would otherwise delay acquisition or capture. In oneexemplary embodiment, the light field data acquisition device maycapture these “analysis” images at regular intervals occurring at afrequency higher than once a second while in a mode that allows foracquisition or capture.

Conventional Auto-Focus Followed By Light Field Sensor Plane Adjustment:In another embodiment, the light field data acquisition device accordingto at least certain aspects of certain embodiments of the presentinventions and/or which may implement certain aspects of certainembodiments of the present inventions, includes auto-focus (for example,a conventional auto-focus system available on conventional cameras).(See, for example, FIGS. 5M and 5N). In this regard, the auto-focussystem may be configured or modified to accommodate implementation witha microlens array of the light field data acquisition device 10 toprovide a device having auto-focus functionality. In this embodiment,the light field data acquisition device 10 may configure the focusparameters or characteristics such that the light field sensor 14 (andthe corresponding light field sensor plane 24) may be located orpositioned a predetermined distance from the focal plane of the subjectof interest in the optics of the device (subject's in-focus focalplane). In one embodiment, the predetermined distance may bemathematically or empirically determined, calculated or characterizedfor different configurations (for example, different ISO settings,aperture, focal length, and/or any other parameters that affect orimpact the performance of the auto-focus system) of the light field dataacquisition device 10 to provide or produce a database or look-uptable(s) that correlates and/or map(s) predetermined configuration ofthe light field data acquisition device 10 to the location, movementand/or displacement of the optics and/or sensor.

In operation, control circuitry of the light field data acquisitiondevice 10 may employ the database or look-up table(s) to move, adjustand/or locate light field sensor 14 a predetermined, selected or desireddistance (e.g., ε) from the subject's in-focus image plane. In responseto instructions from the control circuitry, the physical focus of thelight field data acquisition device 10 moves, adjusts and/or locates thelight field sensor plane 24 before (for example, immediately orsubstantially immediately before) acquisition or capture of light fieldimage data or information by the light field sensor 14 of the lightfield data acquisition device 10.

With reference to FIG. 11A, in one embodiment the light field dataacquisition device 10 may start with an initial separation between lensplane 18 and light field sensor 14. Using an autofocus mechanism (forexample, using a Dedicated Auto-focus sensor presented herein), theseparation between light field sensor 14 and lens plane 18 may bechanged (for example, by mechanically moving the light field sensorrelative to the optics) so that the subject may be in a predeterminedfocus (for example, in optical focus). Before (for example, immediatelyor substantially immediately before) acquisition or capture of lightfield image data, light field sensor 14 may be offset by a predeterminedamount (for example, to place the subject in a predetermined focusrelative to light field sensor 14) relative to the lens plane. Notably,the location of the light field sensor relative to the lens plane may bechanged by any means, including but not limited to all systems andtechniques presented herein.

Notably, the control circuitry may be implemented via a plurality ofdiscrete or integrated logic, and/or one or more state machines, specialor general purpose processors (suitably programmed) and/or fieldprogrammable gate arrays (or combinations thereof). Indeed, allcircuitry (for example, discrete or integrated logic, state machine(s),special or general purpose processor(s) (suitably programmed) and/orfield programmable gate array(s) (or combinations thereof)) to performthe techniques, operations and/or methods, consistent with inventionsdescribed and/or illustrated herein, are intended to fall within thescope of the present inventions.

In those embodiments where the light field data acquisition deviceincludes multiple discrete, fixed-focus positions, the control circuitrymay control and/or adjust the location of the light field sensor 14between one of the plurality of discrete, fixed-focus positions (forexample, the three position exemplary embodiment of FIG. 10A or fourposition exemplary embodiment of FIG. 10C) based on one or moreconsiderations. Here, the control circuitry may employ any technique nowknown or later developed—for example, the control circuitry maydetermine the location of the light field sensor 14 (and correspondinglythe light field sensor plane 24) based on:

-   -   the closest fixed-focus position; and/or    -   the fixed-focus position that provides, allows and/or permits        refocusing the furthest towards infinity, such that the imaged        object's in-focus plane is within the refocusing range of that        fixed-focus position.

Dedicated Auto-Focus Conventional Sensor: In one embodiment, the lightfield data acquisition device according to the present inventions mayemploy an off-the-shelf auto-focus system. With reference to FIGS. 5M,5N and 11B, in this embodiment, the light field data acquisition devicemay employ a plurality of image sensors, for example, a first imagesensor for capturing or acquiring light field image data or informationin a first resolution (for example, a high resolution), and a secondimage sensor to implement or perform auto-focus and capture or acquire asecond image data or information set. The second sensor may capture oracquire image data or information at a second resolution (for example, alower resolution relative to the first resolution). In this embodiment,the auto-focus sensor may not have a microlens disposed in frontthereof. Notably, a multiple sensor configuration according to thepresent inventions may be advantageous in a digital single lens reflex(DSLR) camera or system type configuration or architecture.

With reference to FIG. 11B, in one embodiment, light field dataacquisition device 10 may contain light field sensor 14 and aconventional image sensor used for auto-focus. In the figure, a lens ormirror may be disposed between the optics and the light field sensor todivert some or all of the incoming light rays onto the sensor used forauto-focus. The information gathered from the auto-focus sensor may beused and/or analyzed by applying focus metrics (for example, bymeasuring localized image contrast). These focus metrics may used toguide the focusing system (for example, by adjusting the location of theoptics relative to the light field sensor). In some embodiments, theoptical path length from the optics to the auto-focus sensor may be thesame as or substantially the same as the optical path length from theoptics to the light field sensor. In other embodiments, the optical pathlength from the optics to the auto-focus sensor may be different from,but fixed relative to, the optical path length from the optics to thelight field sensor 14. In a specific embodiment, the optical path lengthfrom the optics to the auto-focus sensor (OP_(AF)) may be fixed relativeto the optical path length from the optics to the light field sensor 14(OP_(LF)) in the following manner:OP _(AF) =OP _(LF)−εNotably, in this specific embodiment, the subject of the autofocus maybe placed at the far end of the designated refocusing range.

Light Field Contrast Auto-Focus: Certain conventional auto-focus systemsuse image contrast to determine what is in focus or is not in focus. Byway of background, at a high level, conventional auto-focus systemstransition (typically over approximately a second) the lens' focus overits focus range, and at each position the contrast of regions of theobserved image is computed. The focus depth that produces the highestcontrast is assumed to be the position at which the imaged subject is infocus.

In an exemplary embodiment of the present invention, a light field dataacquisition device 10 according to the present inventions employs such aconventional auto-focusing system (for example, a system that maximizeslocalized image contrast). (See, for example, FIGS. 5M and 5N). In onesuch embodiment, the modification of the optical signal on the lightfield sensor plane 24 due to the microlens array causes the conventionalcontrast auto focus system to converge the physical auto focus settingonto a position that is approximately equal but slightly offset closerto the lens plane relative to the optical focus position, which is thetypical convergence position achieved in a conventional camera. In someembodiments of the present inventions which implement a conventionalauto-focusing system, this relative offset may be a fixed distance, forexample a multiple of the width of a microlens of microlens array 15,which may be predetermined mathematically or empirically throughmeasurement of the behavior of such a light field data acquisitiondevice. In some embodiments of the present invention, this predeterminedoffset is used to reposition the location of light field sensor 14 ofthe light field imaging system relative to the optical focus positionafter achieving or providing convergence of the physical auto focussetting onto a position that is approximately equal but slightly offsetcloser to the lens plane relative to the optical focus position. Forinstance, in one exemplary embodiment, after convergence of the physicalauto focus setting in connection with such a position, light fieldsensor 14 may be repositioned to a distance of ε closer to the lensplane than the optical focus plane (for example, via mechanical system28) to provide a predetermined range of refocusing (for example,maximal) from the subject at the closest part of the range, andextending beyond. With reference to FIG. 11C and continued reference tothis embodiment, the light field data acquisition device 10 may startwith an initial separation between the lens plane 18 and the light fieldsensor 14. Using an autofocus mechanism (for example, using Light fieldcontrast auto-focus presented herein), the separation between the lightfield sensor 14 and lens plane 18 may be changed (for example, bymechanically moving the light field sensor relative to the optics) sothat the subject may be in a predetermined focus (for example, inoptical focus). Before (for example, immediately or substantiallyimmediately before) acquisition or capture of light field image data,the light field sensor 14 may be offset by ε relative to the optics,such that the separation is decreased by ε. Notably, the location of thelight field sensor 14 relative to the lens plane 24 may be changed byany means, including but not limited to all systems and techniquespresented herein.

Image Contrast Metric for Light Field Data Acquisition Device: In aconventional image, high image contrast often indicates in-focus regionsand low image contrast tends to indicate out-of-focus regions. In oneembodiment, a light field data acquisition device, according to certainaspects of the present inventions, employs an auto-focusing system(having processing and/or control circuitry) that determines the degreeof focus of predetermined regions based on alternate metrics. (See, forexample, FIGS. 5M and 5N). That is, in one embodiment, the auto-focussystem according to an embodiment of the present inventions may assess,determine and/or analyze an image corresponding to light field imagedata (acquired by the light field data acquisition device) and, theauto-focus system, based thereon, may determine the degree of focus of aregion. For example, the light field data acquisition device having theauto-focus system may determine the degree of focus based on: a highcontrast within a light field disk image indicates an out-of-focusregion, and a low contrast within a light field disk image indicates anin-focus region. As such, in one embodiment, the physical auto-focusingsystem implemented in a light field data acquisition device according tothe present inventions employs a metric that is different from metricsemployed by conventional auto-focusing systems.

With references to FIGS. 17A and 17B, these figures illustrate thevariation in contrast in light field disk images as focus changes. InFIG. 17A, the subject is out of focus, resulting in individual lightfield disk images with relatively higher variation of pixel colors. Incontrast, FIG. 17B displays the same area with the subject in focus,resulting in individual light field disk images with relatively lowervariation of pixel colors. In one embodiment, the light field dataacquisition device may acquire a series of light field image datasamples where the focus may be varied with each sample. For each disk ineach sample, a focus metric may be applied (for example, the statisticalvariance or standard deviation of pixel values) to each light field diskimage. These focus metric scores may be aggregated (for example, summedfor all light field disk images in a sample) for each sample, and may beused to select a depth of predetermined focus (for example, by using thelowest aggregate variance to determine a depth of sharpest focus).

Notably, in one embodiment, circuitry in the auto-focus system of thelight field data acquisition device of the present inventions maydiscard or ignore information or data associated with edges of disks ofthe microlens array when determining or analyzing a light field contrastof one or more portions of a light field image. In this regard, in thisembodiment, the auto-focus system of the light field data acquisitiondevice (when determining whether or not an object or subject is in apredetermined focus) may improve or enhance the quality of a light fieldcontrast physical auto-focus system by disregarding or ignoring pixelscorresponding to disk-edges and concentrating on data or informationassociated with other pixels of the sensor—i.e., pixels corresponding tonon-edges.

Scene-Based Light Field Auto-Focus: In addition to light field contrastmetrics, or in lieu thereof, an auto-focus system according to thepresent inventions may employ a scene-based light field metric. In thisregard, given that a light field data from a light field dataacquisition device is capable of refocusing an image over a range ofscene depths (after acquisition of the image data), in one embodiment,the light field data acquisition device employs an auto-focus systemthat determines the depth that is in a predetermined focus (for example,in-focus) at one or more spatial locations in a given image or scene,and based thereon determines, selects and/or identifies a focus positionof the optics of the light field data acquisition device such that one,some or all (or as many as practically possible) of those depths arewithin the refocusable range of the acquired or captured light fieldimage data. Based the determined focus position, the auto-focus systemmay configure the optics (and/or the position of the sensor andmicrolens array) of the light field data acquisition device.

For example, in an illustrative example, where a scene contains aplurality (for example, two) subjects and/or objects at differentdistances from the light field data acquisition device, the auto-focussystem may determine, select and/or identify a focus position betweenthe two subjects and/or objects. Indeed, light field data acquisitiondevice 10 may determine the optimal focus position is in between suchsubjects and/or objects, to enable after-the-fact refocusing to bringeither or both into a predetermined focus—or to bring both as close aspossible to the a predetermined focus or focuses. In one embodiment, theauto-focus system of the light field data acquisition device maydetermine and configure the focus position of the optics (and/or theposition of the sensor and microlens array) to provide the same orsubstantially the same predetermined focus for plurality of thesubjects.

Notably, auto-focus systems according to the present inventions mayemploy any technique now known or later developed—for example, wherethere exists a plurality or list of in-focus image plane locations (fordifferent spatial parts of the scene), the focus position of the lightfield data acquisition device may be determined or based (at least inpart) on:

-   -   Use the mean image plane location; and/or    -   Use the image plane location which permits refocusing to the        greatest number of in-focus image locations.

Such a plurality or list of focus positions may be generated, determinedand/or produced by assessing, analyzing and/or determining the focusdepth on a regular grid of image positions, for example by using an M×Mor M×P grid of positions of a pre-image that are shown or provided tothe user or operator (for example, via a user interface having aview-finder or LCD or the like display, for example, provided to theuser or operator during live-viewing) to determine an appropriate,selected and/or predetermined focus. (See, for example, a 3×3 grid inthe exemplary illustration of FIG. 12).

Notably, any technique or method now known or later developed toimplement after-the-shot focusing may also be applied to the imagedlight field pre-shot by a light field physical auto-focus system.

Selection of Microlens Resolution: As noted above, in certainembodiments of the present inventions, the light field data acquisitiondevice may position the optics and/or light field sensor 14 such thatafter-the-fact light field processing (i.e., processing of light fielddata) is capable of refocusing from a predetermined distance relative tothe light field data acquisition device (for example, a minimum closedistance) to another predetermined distance (for example, infinity oroptical infinity). For example, where the framed object is a human face,a field of view is on the order of 15-30 cm by 15-30 cm; under thesecircumstances, other parameters of the light field data acquisitiondevice (for example, sensor size, lens focal length, desired outputresolution and sharpness, F-number, microlens diameter, etc.) may beselected, implemented, determined and/or chosen to provide a refocusingof the light field image data from infinity (or approximately infinity)to a close or predetermined distance which also provides thepredetermined field of view.

For example, if parameters of the light field data acquisition deviceare fixed except for the zoom range of the lens of the optics, and thelight field data acquisition device is to be designed to have a passiveauto focus system such as the one in FIG. 4, then the longest focallength in the zoom range of the lens of the optics may be determined,set and/or implemented by, for example, tightly framing an object (forexample, a human face) at a distance that corresponds to the closerefocus distance. In this regard, the close refocus distance (hence thesize of the field of view that is framed at that distance) may be anincreasing function of the focal length of the optics of the light fielddata acquisition device. Hence, the maximum focal length of the zoomlens system may be constrained to be the maximum number that satisfiedthe framing condition.

In another embodiment, in addition to or in lieu of the focal lengthparameter, another parameter of the light field data acquisition devicethat may be varied based on the predetermined (for example, minimum)close distance is the number of fixed focus positions to use. In oneembodiment, with reference to FIG. 18, the number of fixed-focuspositions may be determined based on a close focus distance and a lightfield sensor 14. In FIG. 18, the locations of the infinity focal planeand the close focus focal plane are separated by a distance determinedby the optics and the close focus distance. In order to obtain fullcoverage of the focus range, the light field data acquisition device mayuse the following number of fixed-focus positions:NumPositions=ROUND-UP(Focal plane separation/(ε₁+ε₂)),Where ROUND-UP is a mathematical function taking one argument andreturning the smallest integer larger than the given argument. Notably,the formula above may construct focus positions such that each potentialfocal depth may be within the refocusing range of one focus position. Insome embodiments, the number of focus positions may be greater to allowoverlapping zones of focus.

In another embodiment, the resolution of the microlens array 15 (forexample, the total number of lens elements in the microlens array) maybe selected, defined, determined and/or chosen to provide, facilitateand/or enable a trade-off between the ability to refocus an image andthe output resolution of computed images. As noted herein, larger valuesof N allow for an increased range of refocusability, but may result inrefocused images with a lower maximum resolution. In some exemplaryembodiments, the light field data acquisition device 10 has a value forN such that 8≦N≦12.

Notably, in addition to selecting, defining and/or choosing themicrolens resolution to provide or enable a predetermined or givenrefocusing capability, or in lieu thereof, the microlens resolution maybe selected, defined, determined and/or chosen to provide, facilitateand/or enable minor focus corrections of images captured using a dataacquisition device with a physical focus system, such as a DSLR. In someembodiments, the light field data acquisition device 10 may employ alight field sensor 14 with more limited refocusing ability (for example,using a smaller number for N) in order to produce higher resolutioncomputed images. In some embodiments, the light field data acquisitiondevice may 10 include manual or automatic focusing mechanisms thatattempt to capture the light field image in sharp optical focus, andprocessing circuitry may in some circumstances compute refocused framesusing the focus of the light field as it was captured or acquired. As anexemplary embodiment, a DSLR style light field data acquisition device10 may include a high resolution sensor (for example, 50 mega pixels to1000 mega pixels) and a high resolution microlens array (for example,containing 1 million to 20 million lens elements). This exemplaryembodiment may comprise a small N (for example, 2≦N≦8). In oneembodiment, light field sensor 14 in the light field data acquisitiondevice 10 may have an N that is between 5 and 7. Notably, some exemplaryembodiments may also perform aberration correction to improve thequality of refocused frames (see, for example, United States PatentApplication Publication 2009/0128669, titled “Correction of OpticalAberrations”, and the provisional applications to which it claimspriority; United States Patent Application Publication 2007/0252074, andthe provisional application to which it claims priority; and Ren Ng'sPhD dissertation, “Digital Light Field Photography”, Stanford University2006, all of which are incorporated here in their entirety byreference),

In other embodiments, it may be desirable to enable dramatic refocusingover a large range of focus (for example, in order to bring a face thatwas originally unrecognizable into sharp focus). In some embodiments, alight field data acquisition device 10 designed for dramatic refocusingmay have an N which is greater the 12.

Determination of Light Field Sensor Placement for Fixed-Focus: In someembodiments, the placement of the light field sensor relative to thefocal point may be determined by calculating and/or determining amaximum distance ε where subjects at or near optical infinity may bebrought into sharp focus based on the properties and/or configuration ofthe light field data acquisition device, and placing the light fieldsensor at that distance ε beyond the focal point relative to the optics.

Notably, in many of the following embodiments, the optical depth offield of the optics with respect to the light field sensor does notinclude optical infinity. For clarity, as a way of illustration, in manyof the following embodiments, if the light field image sensor werereplaced with a conventional imaging sensor placed in the same manner,pictures of subjects at optical infinity would not appear sharp.

Briefly, the selection of ε (and the location of the light field sensor)may be performed such that the light field sensor may be placed themaximum distance beyond the focal point of the optics which allowsprocessing circuitry to compute an image, virtually refocused onto asubject appearing at optical infinity, where that subject appearsvisually sharp to an observer. In comparison to a device where the lightfield sensor is placed in such a manner, in the cases where ε may begreater (and the light field sensor is further from the focal point),images refocused to a subject appearing at optical infinity may appearsoft and/or out of focus to an observer. Further, in cases where ε maybe smaller (and the light field sensor is closer to the focal point),light fields acquired by the device may not be able to focus as sharplyonto subjects close to the device.

In one embodiment, ε (and the location of the light field sensor) may beselected such that the light field sensor may be placed the maximumdistance beyond the focal point of the optics which allows processingcircuitry to compute an image, virtually refocused to a subjectappearing at optical infinity, which appears visually sharp to anobserver, and further, that such image an image may appear substantiallyat least as sharp as an image refocused onto a subject appearing to beat a location “beyond optical infinity”. Briefly, when referring to asubject appearing at a location “beyond optical infinity” and/or“focusing beyond optical infinity”, “beyond optical infinity” may becharacterized as a subject and/or ray source in which the rays appear tobe divergent relative to the optics of the light field data acquisitiondevice. As a way of example, parallel rays passing through a concavelens or lens group prior to passing through optics of the light fielddata acquisition device, would appear divergent relative to the device,as if they had originated from a subject “beyond optical infinity”. In asimilar manner, a refocused image computed using a virtual plane offocus that is located closer to the optics than the focal point, may becharacterized as “focusing beyond optical infinity”.

Notably, the appearance of visual sharpness may vary as the size of thecomputed refocused images is changed. As a result, images computed atsmaller resolutions (for example, 640×480 pixels) may appear sharp whenthe same image computed at a larger resolution (4000×3000 pixels) wouldnot.

As noted herein, selection of the appropriate value(s) for ε may varybased on the intended output resolution of refocused images. In someembodiments, the following formula may be used:

-   -   Let W_(MLA) equal the width of the microlens array, in number of        lens elements    -   Let W_(output) equal the width of the refocused image, in number        of pixels        Determines ε such that:

$ɛ = \frac{C*N_{usable}*W_{MLA}*F\#_{optics}*m}{W_{output}}$where C is a constant. In some embodiments, 0.7≦C≦3. In theseembodiments, minimum and maximum values for ε are established based onthe intended resolution of images computed from light fields captured bythe device.

In another embodiment, ε (and the location of the light field sensor)may be determined for devices where the sharpness of the computed imagesmay not be limited by the resolution of the computed images (forexample, when the resolution of the computed images is more thansufficient to accurately depict all details that may be captured). Inthese embodiments, ε may be selected without regard for the outputresolution of computed images and may be based on the formula previouslypresented herein:ε=KN _(usable) m(F# _(optics))In some embodiments, a minimum value for ε may be established at thepoint where the diameter of the optical blur on the light field sensorfrom rays originating at optical infinity is at least m (the diameter ofa microlens). In these embodiments, b≧m, which, solving for K, leads to:b≧mε=b(F# _(optics))ε=KN _(usable) m(F# _(optics))Thus,K≧1/N _(usable)Notably, as the diameter of the blur introduced by misfocus at thisseparation equals the size of a microlens diameter, the optical depth offield of the optics with respect to the light field sensor may beconsidered to end nearer to the device than optical infinity. In otherembodiments, a minimum value for ε may be established at the point wherethe diameter of the optical blur on the light field sensor from raysoriginating at optical infinity is at most N_(usable)*m, which may allowrefocusing with detail maintained to approximately the resolution of themicrolens array. In these embodiments, K≦1.0. In one exemplaryembodiment, 1/N_(usable)≦K≦1.0. In a specific exemplary embodiment,K=0.6.

In another embodiment, ε (and the location of the light field sensor)may be determined based on the output resolution of computed images andempirical analysis of the resolution of computed refocused images. Inthis embodiment, the resolution of the image in lines per picture height(LPH) may be measured across a range of values for ε (for example, bytaking pictures of a resolution chart (see, for example, InternationalStandards Organization, Standard 12233 (www.iso.org) or the like)). Inthis manner, ε may be selected as the largest or near largest value suchthat an image refocused onto a subject located at or near opticalinfinity would appear sharp at the output resolution. In one embodiment,the measurable resolution in LPH of the scene refocused to opticalinfinity may be between 0.5 and 0.8 times the height of the computedimage size, in pixels. Briefly, lines per picture height (LPH) may becharacterized as the number of distinct lines that may be distinguishedacross a distance equal to the height of the picture. One way ofmeasuring LPH may be to acquire an image of a specifically designedresolution chart (see, for example, FIG. 21), and then determine byinspection the highest number of lines that can be visuallydistinguished.

In another embodiment, ε (and the location of the light field sensor)may be determined based on the maximum resolution of computed images andthe empirical resolution of computed refocused images. Briefly, in thiscontext the maximum resolution is the maximum resolution in LPH for arefocused image sharply focused at any value for ε. In this embodiment,the resolution in LPH may be measured across a range of values for ε(for example, by using a resolution chart (see, for example,International Standards Organization, Standard 12233)). In this manner,ε may be selected as the largest or near largest value such that animage refocused onto a subject located at or near optical infinity wouldmaintain a measurable resolution in LPH of a fraction of the maximumresolution.

Notably, in cases where the measurable resolution may be limited by thesize of the computed images, the height, in pixels, of the computedimage may be considered the maximum resolution (for example, the maximumpossible resolution in LPH of an image 640×480 pixels may be considered480).

Generally, as ε increases, the measurable resolution for an image,focused sharply and computed at a virtual focus depth a distance ε fromthe light field sensor plane, decreases. In some embodiments, it may bedesirable that an image computed to focus on a subject located at ornear optical infinity appears sharp to the viewer. In these embodiments,the location of ε may be set such that the measurable LPH of imagesrefocused to subjects located at optical infinity is a minimum fraction(for example, one half) of the maximum resolution. Notably, in theseembodiments, an upper bound may be determined for E. Furthermore, undernormal circumstances, the ability of the light field data acquisitiondevice to enable sharp refocusing onto subjects beyond optical infinitymay not be needed. Indeed, by making ε too small, the device may losethe ability to sharply focus on subjects near the device (for example,at the close focus distance). In some embodiments, ε may be set suchthat the measurable LPH of images refocused to subjects located atoptical infinity is no greater than a fraction (for example,four-fifths) of the maximum measurable LPH. Notably, in theseembodiments, a lower bound may be determined for ε. In one embodiment,measurable resolution in LPH of the scene refocused to optical infinitymay be between 0.5 and 0.8 times the LPH of the maximum resolution forthe device.

As noted herein, in some embodiments ε may be determined in order toallow a range of refocusing from infinity to a close focus distance.With that in mind, in one embodiment, one of the predetermined distances(for example, the minimum close distance) is determined and fixed (forexample, temporarily or permanently, and/or during manufacture, atstart-up) by framing an object (for example, a human face). In oneembodiment, ε may be determined, where:

-   -   Let L_(focal) be equal to the focal length of the main lens, in        millimeters    -   Let H_(sensor) be equal to the height of the sensor, in        millimeters    -   Let H_(subject) be equal to the height of the subject, in        millimeters        Such that:

$ɛ \geq {\frac{1}{2*( {\frac{1}{L_{focal}} - \frac{H_{sensor}}{L_{focal}*H_{subject}}} )} - \frac{L_{focal}}{2}}$

In this manner, ε may be selected such that the light field sensor planeis placed evenly between the focal plane of the optics and the plane offocus for an object located at the selected close focus distance. In anexemplary embodiment, the chosen close subject is a human face, henceH_(subject) is chosen to be approximately 300 millimeters. In anotherexemplary embodiment, the chosen close subject is a standing human,hence H_(subject) is chosen to be approximately 2000 millimeters.

Zoom Lens with Constant Exit Pupil: Some designs for main lenses havethe characteristic that the exit pupil changes its apparent size and/orlocation as the physical focus and/or zoom changes. Utilizing such amain lens design in a light field data acquisition device, may affectthe geometry of the captured light field as a function of physical focusand/or zoom. Under these circumstances, there are various approachesthat the light field data acquisition device of certain aspects of thepresent inventions may implement to address this phenomenon, including:

-   -   Calibrate the light field processing such that at each        focus/zoom position, the processes are able to model the        geometry with sufficient accuracy as a function of the        focus/zoom position; and/or    -   Use a main lens design in which the exit pupil's size and        position does not change with zoom and/or focus. One example of        this design is to have static (non-moving) lens elements between        the aperture of the light field data acquisition device and the        sensor thereof, with all moving lens elements between the        aperture and the scene (outside World).

In some embodiments, the light field data acquisition device may becalibrated at each focus/zoom position to create a geometric model ofthe light field disk images as they project onto the imaging sensor.This calibration may be performed in the following manner:

-   -   Acquire or capture a light field image for that zoom/focus        position of an evenly illuminated or low contrast scene (for        example, a wall). It may be desirable to capture this image        using a small aperture.    -   Determine the geometric parameters, starting with known values,        including pixel size, microlens size, pixel pattern, and        microlens grid pattern.    -   Find the center of an initial light field disk image, by        maximizing the value of a search function.        -   In one search function, a series of potential microlens            centers are tested (for example, a grid spaced at ¼ pixel            increments).        -   For each potential center, a spatial filter may be applied            (for example, a cubic filter with a positive area equal to            the expected area of a light field disk image.        -   The potential center with the largest score may be selected            as the center of the microlens element.    -   Estimate the location of a neighboring light field disk center        using the previously determined center and the known parameters        (pixel size, microlens size, and grid patterns).    -   Determine the center of the neighboring light field disk center        using the estimated center and by maximizing the value of a        localized search function (for example, as described above).    -   Continue finding light field disk centers by estimation and        search function (as in the previous 2 steps) until enough        centers are located (for example, all centers or all centers        inside a 500 pixel border).    -   Use the centers located to create a geometric model that is a        good fit for the determined data.        -   In one embodiment, the light field disk image grid pattern            (for example, rotation and separation of light field disk            image grid coordinates) and offsets (for example, the X and            Y offset from the center pixel to a light field disk image            grid coordinate).            In some embodiments, the geometric model determined from            calibration is used when computing refocused images. (See,            for example, Ren Ng's PhD dissertation, “Digital Light Field            Photography”, Stanford University 2006, U.S. Provisional            Application Ser. No. 61/170,620, entitled “Light Field            Camera Image, File and Configuration Data, and Method of            Using, Storing and Communicating Same”, filed Apr. 18, 2009.

Device Optics for Maximizing Captured Directional Resolution Across Zoomor a Predetermined Portion Thereof: A typical zoom lens property is tohave the lens' aperture vary with the focal length. For example, a lensmay be f/2.8 at its 35 mm zoom position, and f/5.6 at its 200 mm zoomposition. In a light field data acquisition device, the nature of thelight captured relates to the relationship between the microlensF-number and the main lens F-number, as is illustrated in FIG. 13.

With reference to FIGS. 3A and 13, where the F-number of main lens 20 ishigher than the F-number of a microlens (x,y) of array 15, then the“disk images” that appear on and are captured by sensor 16 are smaller,and in effect the directional resolution of the captured light field isreduced. This is shown in an illustrative light field image, in whichFIG. 14A shows “full-sized” disk images, and FIG. 14B shows smaller diskimages; the unshaded areas correspond to non-illuminated sensor pixelsthat are wasted or unused. In some embodiments, the usable number ofpixels across each microlens disk image, N_(usable), that may be usedfor processing may be reduced using the following:N _(usable) =N*F# _(MLA) /F# _(optics)Correspondingly, where the F-number of the main lens 20 is smaller thanthe F-number of a microlens of array 15, then the “disk images” thatappear on and are captured by sensor 16 are larger, and in effectoverlap. This is shown in an illustrative light field image, in whichFIG. 14C shows overlapping disk images. In some embodiments withoverlapping disk images, the usable number of pixels across eachmicrolens disk image, N_(usable), is given by the following:N _(usable) =N*(2.0−F# _(MLA) /F# _(optics))

In a further embodiment, N_(usable) may be reduced by the width of oneor two pixels. Main Lens with F-Number Independent of Zoom: With theaforementioned in mind, in one embodiment, the light field dataacquisition device, according to certain aspects of the presentinventions, may include a main lens which provides, includes and/ormaintains the same or substantially the same effective aperture (e.g.constant exit pupil) across its zoom range, or a predetermined portionthereof, and/or which correlates or matches to the F-number of amicrolens of the microlens array. With reference to FIGS. 3A and 15, inone embodiment, light field data acquisition device 10 includes amechanism to stop down the aperture of main lens 20 for shorter focallengths in order to match the highest F-number of main lens 20 acrossits range of zoom and/or range of focus.

In another embodiment, the light field data acquisition device includesa main lens having a constant or substantially constant F-number (forexample, a maximum F-number). In one embodiment, the lens elements ofthe optics between the aperture of the light field data acquisitiondevice and the sensor-side of the lens do not vary with zoom position,focus and/or the image of the aperture as seen from the perspective ofthe sensor does not vary. Briefly, there are commercially available zoomlenses that maintain a constant or substantially constant maximumF-number across a range of zoom positions. In some exemplary embodimentsof the present aspect of the inventions, the design of the acquisitionsystem selects such lenses, or adopts such design for lenses, thatmaintain a constant or substantially constant maximum F-number across arange of zoom positions, Notably, any lens design currently known orinvented in the future, which maintains a constant or substantiallyconstant maximum F-number across a range of zoom positions, is intendedto fall within the scope of the present aspect of the inventions.

In addition, there are commercially available zoom lenses that maintaina constant or substantially constant exit pupil across a range of zoomand/or optical focus positions. In some exemplary embodiments of thepresent aspect of the inventions, the design of the acquisition systemselects such lenses, or adopts such design for lenses, that maintainsuch a constant or substantially constant exit pupil across a range ofzoom and/or optical focus positions. Notably, any lens design currentlyknown or invented in the future, which maintains a constant orsubstantially constant exit pupil across a range of zoom and/or opticalfocus positions, is intended to fall within the scope of the presentaspect of the inventions.

In sum, by maintaining a constant exit pupil in ways such as these, orby employing other techniques or structures to provide a constant exitpupil, the captured disk images may be “full-sized” or more full-sizedacross all zoom or focus positions and/or a portion thereof.

Live View

Notably, the light field data acquisition device may include processingcircuitry 38 to compute images (using light field data or informationacquired, sampled, sensed and/or obtained by light field sensor 14) forreal-time (or near real-time) display to the user or operator via userinterface 36. (See, for example, FIG. 3B). In one embodiment, data whichis representative of the images is provided to a display of userinterface 36 on data acquisition device 10, in a real-time or nearreal-time manner, during subject composition and before image data orinformation acquisition for the final image. This may be referred toherein as a “live-view” mode.

The live-view processing may be implemented using any light fieldprocessing technique now known or later developed, including, forexample, a fully general high-quality refocusing computation featuringscene analysis to determine what focus depth and aperture to use for thedisplayed image. All such light field data acquisition and/or processingtechniques to implement live viewing are intended to fall within thescope of the present inventions.

For example, in one embodiment, processing circuitry 38 of light fielddata acquisition device 10 generates a live-view image using a portionof the data or information collected, acquired and/or sampled by lightfield sensor 14. In this regard, light field data acquisition device 10may down-sample to accelerate generation of image data which isrepresentative of the images to be provided to user interface 36 (forexample, a display) during the live-view process. For example, in oneembodiment, light field sensor 14 of light field data acquisition device10 (or circuitry associated therewith) may sub-sample and/or down-samplethe light field image by storing, reading, sampling and/or obtaining asubset of pixels associated with each disk image, optionally combiningor filtering their values, and aggregating the light field data inand/or generating an image array in order to construct an image which islower-resolution relative to images generated using more or all of thelight field data acquired by light field sensor 14. The lower resolutionimage (which may be generated more rapidly than a higher resolutionimage using more of the light field image data) may be provided to theuser interface 36 for display to the user or operator during thelive-view process.

In one exemplary embodiment, the subset of pixels of each disk may bethose pixels near the center regions of disk images, for example, thecenter 50% of the pixels. In another exemplary embodiment, withreference to FIG. 20, the pixel nearest to the center of each microlens“disk image” may be selected and a small live-view frame may be computedwhere the value for each pixel in the live view frame is the value ofthe pixel nearest the center of the corresponding “disk image”. In thisway, this technique may produce a low-resolution image with a depth offield that extends over the entire refocusing range. Notably, where thelight field data acquisition device includes a fixed-focus configurationor architecture (for example, a refocusing range from some closedistance to infinity), the depth of field of the generated imageapproximately extends from the same close distance to infinity. Indeed,the sub-sampled image may be re-sampled into a desired output resolutionthat matches the display for live-view.

Narrow Depth of Field by Down-Sampling One or More Entire Disks: Inanother embodiment, the light field data acquisition device maydown-sample pixels across entire disks into, and aggregate the resultingdown-sampled pixels into a 2D array to construct an output image.Moreover, in certain embodiments, partially-illuminated disk edge pixelsmay be discarded or scaled up to full intensity. This technique mayprovide or yield a live-view image that has better signal to noisecharacteristics.

In another exemplary embodiment, the light field image sensor maycontain readout mode(s) in order to allow a higher frame rate, togenerate preview frames, to generate frames for video recording, and/orto generate narrow depth of field frames directly from sensor data.Briefly, conventional image sensors may contain a plurality of readoutmodes which may fully sample, sub-sample, and/or combine pixel data tooutput a plurality of image sizes at a plurality of frame rates andgenerally each mode may have differing speeds and power requirements.Depending on how the conventional digital camera is being used (forexample, acquiring a still picture at a certain resolution, displayinglive view, or recording video), a different readout mode may be used. Insome embodiments, the light field sensor and microlens array may bedesigned and/or manufactured in tandem to allow precise alignment ofmicrolens elements to the sensor readout mode(s). In another embodiment,the light field sensor may allow sufficient readout mode configurationflexibility to be tuned after manufacturing and/or image data mayundergo additional post-processing before or after the read out. Inanother embodiment, the light field sensor may have a readout mode thatreads one or multiple pixel data from all or alternating rows and all oralternating columns of light field disk images which later may undergoeither (a) combining the pixels by color (e.g. R is combined with R, Gwith G, B with B), resulting in reduced resolution of sensor readoutpattern (e.g. Bayer mosaic) or (b) combining the pixels (e.g. 2 greenpixels, 1 red and 1 blue) to create single RGB, YUV or any other colorimage representation format or (c) combining the pixels to create agrayscale output image (e.g. luminance component only) or (d) data isreadout without further down-sampling (i.e. without combining pixels),maintaining sensor pattern output (e.g. Bayer mosaic), which may containless pixels than the full sensor pattern. In one specific embodiment,and with reference to FIG. 19A, the sensor 16 may have a readout modethat may output a single RGB or YUV value for each light field diskimage, combining the four pixel values nearest the center of each lightfield disk image (for example, 2 green pixels, 1 red and 1 blue). Inanother specific embodiment, the light field sensor may have a readoutmode that reads a single pixel value (for example, R, G or B) andcombines values for 4 neighboring light field disk images to create anRGB or YUV value. In another embodiment, with reference to FIG. 19B, thelight field sensor may have a readout mode that reads out the fourpixels values nearest the center of each light field disk image (forexample, 2 green pixels, 1 red and 1 blue), and then combine thisreadout without combining pixel values, into another Bayer mosaicpattern that may contain less pixels than the full sensor pattern.Indeed, there are many readout modes that may be desirable when readingfrom a light field sensor, and any readout mode now known or laterdeveloped that couples a readout pattern to the pattern of light fielddisk images is intended to fall within the scope of the presentinventions.Automatic Focusing after the Shot

In another aspect, the present inventions are directed to system 100including post-processing circuitry 102 that automatically computes oneor more images for display after acquisition of the light field imagedata (i.e., after the shot is taken). (See, for example, FIG. 3F,wherein light field data acquisition device 10 may be or include any ofthe embodiments described and/or illustrated herein). In one embodiment,light field data acquisition device 10 acquires, captures and/or sampleslight field image data using, for example, a fixed or simplifiedphysical focus, and, post-processing circuitry 102, implementing one ormore after-the-shot focusing techniques, determines one or more imagesto compute and/or generate. Thereafter, the one or more images may bedisplayed to a user or operator, for example, via a display of, forexample, user interface 36 or external display (for example, through adata output port on the data acquisition device or system). (See, forexample, FIGS. 3B-3E).

Notably, in addition thereto, or in lieu thereof, system 100 may storethe light field data and/or the processed data (for example, therefocused data) which is representative of the one or more images may bestored in or saved to internal or external memory (for example, externalstorage such as a FLASH memory card—see, for example, FIG. 3E). Thesystem 100 may store the “raw” refocusable light field data (as outputfrom light field sensor 14) and/or a representation thereof (forexample, a compressed refocusable light field data, multiple light fielddata (for example, a plurality images of the same scene, each imagebeing refocused at different a focal plane) and/or combinationsthereof). Moreover, the system 100 may store the light field data orrepresentations thereof (for example, one or more images that aregenerated using the light field data) using one or more of theembodiments of U.S. Provisional Patent Application Ser. No. 61/170,620,filed Apr. 18, 2009, entitled “Light Field Camera Image, File andConfiguration Data, and Method of Using, Storing and CommunicatingSame,” which, as noted above, is incorporated herein in its entirety byreference. All permutations and combinations of data storage formats ofthe light field data or representations thereof (for example, one ormore images that are generated using the light field data) are intendedto fall within the scope of the present inventions.

In another embodiment, system 100 may generate a plurality of images,via after-the-shot focusing and acquired or collected light field dataor information, and thereafter automatically generate and produce ananimation thereof to, for example, the user or operator, via a displayon the light field data acquisition device or external display (forexample, through a data output port on the device or system, and/or datanetworks and the internet). (See, for example, FIGS. 3E and 3F). In oneembodiment, system 100 may generate an animation that adjusts focusbetween automatically or manually designated subjects of interest. Foreach subject of interest, the system adds a transition animation from aninitial focus depth to the focus depth of the subject of interest,pauses for a time, and transitions to the next subject of interest.

In yet another embodiment, system 100 may generate a plurality ofimages, using after-the-shot focusing, and use the plurality of imagesto automatically generate a slideshow of the plurality of images. Theslideshow may contain still images generated and/or produced usingsimilar modules. As discussed above, the images may be provided to aninternal and/or external display (for example, an LCD or the like on thelight field data acquisition device) and/or internal and/or externalmemory (for example, FLASH memory card). Indeed, the display and/ormemory may display or contain animations similar to the animationsmentioned above, and, in certain embodiments, may combine images andanimations from multiple light fields to produce new images andanimations. FIG. 16 illustrates, in a block diagram manner, certainaspects of the plurality of the embodiments of system 100, includingafter-the-shot focusing embodiments, techniques and/or operationsperformed thereby. Notably, system 100 may employ any technique nowknown or later developed to implement any of the after-the-shot focusingembodiments. For example, system 100 (for example, light field dataacquisition device 10 or post-processing circuitry 102) may implementscene analysis to compute or provide one or more images. In this regard,the data acquisition device 10 may implement:

-   -   Scene analysis to perform automatic focus selection; and/or    -   Scene analysis to perform automatic aperture (i.e., Depth of        Field) selection; and/or    -   Scene analysis to perform automatic tilt/shift selection.

In one embodiment, system 100 (for example, light field data acquisitiondevice 10 or post-processing circuitry 102 may implement the followingtechnique:

-   -   generate a stack of images that are focused at different scene        depths from, for example, a minimum close distance to infinity;    -   thereafter, analyze each image in order to compute a measure of        importance of the subjects contained in the image; and    -   thereafter, output the image that has the highest measure of        importance.        Notably, a variation on this approach outputs an image that is a        result of compositing the high-importance regions from a number        of frames.

In addition to scene analysis, or in lieu thereof, system 100 (forexample, light field data acquisition device 10 and/or post-processingcircuitry 102) may perform light field scene analysis to implement anyof the after-the-shot focusing embodiments. Indeed, any light fieldscene analysis technique now known or later developed is intended tofall within the scope of the present inventions. For example, the lightfield data acquisition device or system may implement:

-   -   Face-detection: identify where in the captured image faces are        (for example, using a suitably configured Haar cascade        classifier). In general, identify where human-related features        are (e.g.) smile detection, etc; and/or    -   Face-recognition: identify presence and location of known        individuals in the captured image, potentially using a database        that contains computed characteristics of known individuals'        faces. Any facial recognition method now known or later        developed may be used, including principal component analysis        (PCA) or elastic bunch graph matching (EBGM); and/or    -   Focus analysis: for a given image region (e.g., a face)        determine where it is in focus; and/or    -   Focus analysis: for a given image region at a certain focus        depth, determine how in-focus (or the degree of focus) the image        or region thereof is.

In another embodiment, system 100 (for example, light field dataacquisition device 10 or post-processing circuitry 102) may assessand/or determine the focus and depth of field based on evaluatingwhether the image includes one or more faces and—if there are faces inthe image, choose the focus depth to make or bring one or more of thefaces (for example, the largest face) into a predetermined focus (forexample, in-focus); otherwise extend the depth of field as far aspossible.

Alternatively, system 100 (for example, light field data acquisitiondevice 10 or post-processing circuitry 102) may assess and/or determinethe focus and depth of field based on evaluating whether there are aplurality of faces and—if so, select or choose the focus depth that isin the middle of the closest and furthest faces, and extend the depth offield only as far as is required to bring all faces into a predeterminedfocus (for example, in-focus); otherwise, extend the depth of field (forexample, as far as possible).

Notably, any method or technique for ranking the contents of a scene interms of importance, and then choosing the focus and depth of fieldbased on the importance and focus depths of the scene's contents, may beused. Indeed, several of the analysis methods or techniques disclosed inU.S. patent application Ser. No. 12/622,655 (“System of and Method forVideo Refocusing”, filed Nov. 20, 2009). All such methods or techniques,now known or later developed, are intended to fall within the scope ofthe present inventions. For example, computation of the measure ofimportance may be determined as follows:

-   -   A face detection method is used on each image in the stack.    -   If one of more focused faces are detected in an image, then the        measure of importance is the area of the focused faces in the        image.    -   If no faces are detected in the image, then the measure of        importance is the area of the image that is in-focus.        Interaction for User-Guided Focus

In another aspect, the present inventions are directed to a systemincluding circuitry and/or performing techniques that enables theoperator or user to interact with the system 100 (for example, lightfield data acquisition device 10 or post-processing circuitry 102) tocontrol final image processing. In one embodiment, the user or operatormay determine the final output image(s), for example, using theafter-the-shot focusing. The after-the-shot focusing may implement thetechniques described in “Interactive Refocusing of Electronic Images”,U.S. patent application Ser. No. 11/948,901, filed Nov. 30, 2007 (U.S.Patent Application Publication 2008/0131019), which is incorporatedherein in its entirety by reference. For example, system 100 (forexample, light field data acquisition device 10 or post-processingcircuitry 102) may employ one or more of the following:

-   -   Selection of which objects/subjects to focus on, for example by        clicking on the desired portion of the image displayed on a        touch screen on the device; and/or    -   Selection of the depth of field to use; and/or    -   In general, selection of any parameter that may be described        herein as being automatically determined; accordingly, such        parameter may be selected manually and, for the sake of brevity,        such discussion will not be repeated in this context.

Notably, in the exemplary embodiments hereof, the data processing,analyses, computations, generations and/or manipulations may beimplemented in or with circuitry disposed (in part or in whole) in/onthe data acquisition device or in/on an external processing system. Suchcircuitry may include one or more microprocessors, Application-SpecificIntegrated Circuits (ASICs), digital signal processors (DSPs), and/orprogrammable gate arrays (for example, field-programmable gate arrays(FPGAs)). Indeed, the circuitry may be any type or form of circuitrywhether now known or later developed. For example, the signal processingcircuitry may include a single component or a multiplicity of components(microprocessors, FPGAs, ASICs and DSPs), either active and/or passive,which are coupled together to implement, provide and/or perform adesired operation/function/application; all of which are intended tofall within the scope of the present invention.

Further, as mentioned above, in operation, the processing circuitry mayperform or execute one or more applications, routines, programs and/ordata structures that implement particular methods, techniques, tasks oroperations described and illustrated herein (for example, acquiringand/or editing the refocusable light field data and/or generating orrendering output image data corresponding to refocusable light fielddata using one, some or all of the aforementioned acquisition, editingand/or generating techniques). The operations of the applications,routines or programs may be combined or distributed. Further, theprocessing circuitry may implement one or more, or all of suchtechniques in any combination and all permutations; such techniques maybe employed alone or in combination with one or more of the othertechniques of acquiring and/or editing the refocusable light field dataand/or generating or outputting image data corresponding to refocusablelight field data. The techniques, methods and/or applications may beimplemented by the processing circuitry using any programming languagewhether now known or later developed, including, for example, assembly,FORTRAN, C, C++, and BASIC, whether compiled or uncompiled code; all ofwhich are intended to fall within the scope of the present invention.

There are many inventions described and illustrated herein. Whilecertain embodiments, features, attributes and advantages of theinventions have been described and illustrated, it should be understoodthat many others, as well as different and/or similar embodiments,features, attributes and advantages of the present inventions, areapparent from the description and illustrations. As such, the aboveembodiments of the inventions are merely exemplary. They are notintended to be exhaustive or to limit the inventions to the preciseforms, techniques, materials and/or configurations disclosed. Manymodifications and variations are possible in light of this disclosure.It is to be understood that other embodiments may be utilized andoperational changes may be made without departing from the scope of thepresent inventions. As such, the scope of the inventions is not limitedsolely to the description above because the description of the aboveembodiments has been presented for the purposes of illustration anddescription.

As noted above, there are many inventions described and illustratedherein. While certain embodiments, features, materials, configurations,attributes and advantages of the inventions have been described andillustrated, it should be understood that many other, as well asdifferent and/or similar embodiments, features, materials,configurations, attributes, structures and advantages of the presentinventions that are apparent from the description, illustration andclaims. As such, the embodiments, features, materials, configurations,attributes, structures and advantages of the inventions described andillustrated herein are not exhaustive and it should be understood thatsuch other, similar, as well as different, embodiments, features,materials, configurations, attributes, structures and advantages of thepresent inventions are within the scope of the present invention.

Importantly, each of the aspects of the present invention, and/orembodiments thereof, may be employed alone or in combination with one ormore of such other aspects and/or embodiments. For the sake of brevity,those permutations and combinations will not be discussed separatelyherein. Indeed, the present inventions are not limited to any singleaspect or embodiment thereof nor to any combinations and/or permutationsof such aspects and/or embodiments.

As such, the above embodiments of the present inventions are merelyexemplary embodiments. They are not intended to be exhaustive or tolimit the inventions to the precise forms, techniques, materials and/orconfigurations disclosed. Many modifications and variations are possiblein light of the above teaching. It is to be understood that otherembodiments may be utilized and operational changes may be made withoutdeparting from the scope of the present inventions. As such, theforegoing description of the exemplary embodiments of the inventions hasbeen presented for the purposes of illustration and description ofexemplary embodiments. Many modifications and variations are possible inlight of the above teaching. It is intended that the scope of theinventions not be limited solely to the description above.

It should be noted that the term “circuit” may mean, among other things,a single component (for example, electrical/electronic) or amultiplicity of components (whether in integrated circuit form orotherwise), which are active and/or passive, and which are coupledtogether to provide or perform a desired function. The term “circuitry”may mean, among other things, a circuit (whether integrated orotherwise), a group of such circuits, one or more processors, one ormore state machines, one or more processors implementing software, oneor more gate arrays, programmable gate arrays and/or field programmablegate arrays, or a combination of one or more circuits (whetherintegrated or otherwise), one or more state machines, one or moreprocessors, one or more processors implementing software, one or moregate arrays, programmable gate arrays and/or field programmable gatearrays. The term “data” may mean, among other things, a current orvoltage signal(s) (plural or singular) whether in an analog or a digitalform, which may be a single bit (or the like) or multiple bits (or thelike).

It should be further noted that the various circuits and circuitrydisclosed herein may be described using computer aided design tools andexpressed (or represented), as data and/or instructions embodied invarious computer-readable media, in terms of their behavioral, registertransfer, logic component, transistor, layout geometries, and/or othercharacteristics. Formats of files and other objects in which suchcircuit expressions may be implemented include, but are not limited to,formats supporting behavioral languages such as C, Verilog, and HLDL,formats supporting register level description languages like RTL, andformats supporting geometry description languages such as GDSII, GDSIII,GDSIV, CIF, MEBES and any other suitable formats and languages.Computer-readable media in which such formatted data and/or instructionsmay be embodied include, but are not limited to, non-volatile storagemedia in various forms (e.g., optical, magnetic or semiconductor storagemedia) and carrier waves that may be used to transfer such formatteddata and/or instructions through wireless, optical, or wired signalingmedia or any combination thereof. Examples of transfers of suchformatted data and/or instructions by carrier waves include, but are notlimited to, transfers (uploads, downloads, e-mail, etc.) over theInternet and/or other computer networks via one or more data transferprotocols (HTTP, FTP, SMTP, etc.).

Indeed, when received within a computer system via one or morecomputer-readable media, such data and/or instruction-based expressionsof the above described circuits may be processed by a processing entity(e.g., one or more processors) within the computer system in conjunctionwith execution of one or more other computer programs including, withoutlimitation, net-list generation programs, place and route programs andthe like, to generate a representation or image of a physicalmanifestation of such circuits. Such representation or image maythereafter be used in device fabrication, for example, by enablinggeneration of one or more masks that are used to form various componentsof the circuits in a device fabrication process.

1. A light field imaging device for acquiring light field image data ofa scene, the device comprising: optics, wherein the optics includes anoptical path and a focal point, wherein the focal point is associatedwith a focal length of the optics; a light field sensor, located (i) inthe optical path of the optics to acquire light field image data and(ii) at a substantially fixed, predetermined location relative to thefocal point of the optics, wherein the predetermined location issubstantially independent of the scene, and wherein an optical depth offield of the optics with respect to the light field sensor extends to adepth that is closer than optical infinity; a user interface to receivea plurality of user inputs, including first and second user inputs,wherein, in response to the first user input, the light field sensoracquires the light field image data of the scene; and processingcircuitry, coupled the user interface, to generate and output image datawhich is representative of an image, the processing circuitry to:determine a first virtual focus depth of the light field image data,wherein the first virtual focus depth is different from the opticalfocus depth of the light field image data, automatically generate datawhich is representative of a first image of the scene using the lightfield image data, wherein the first image includes a focus whichcorresponds to the first virtual focus depth, output the data which isrepresentative of the first image, and, after outputting the data whichis representative of the first image and in response to the second userinput, determine a second virtual focus depth of the light field imagedata using data which is representative of the second user input,wherein the second user input is indicative of the second virtual focusdepth and the second virtual focus depth is different from the opticalfocus depth of the light field image data, and generate data which isrepresentative of a second image of the scene using the light fieldimage data wherein the second image includes a focus which correspondsto the second virtual focus depth.
 2. The light field imaging device ofclaim 1 wherein the optics is configurable to include a plurality ofdifferent focal lengths having associated focal points.
 3. The lightfield imaging device of claim 2 wherein the substantially fixed,predetermined location of the light field sensor, relative to the focalpoint of the optics, changes in accordance with the focal length of theoptics.
 4. The light field imaging device of claim 1 wherein the opticsincludes a zoom lens system having a plurality of zoom positions.
 5. Thelight field imaging device of claim 4 further including a mechanicalsystem, coupled to the light field sensor, to maintain the light fieldsensor unit at the same fixed, predetermined location relative to thefocal point of the optics for the plurality of the zoom positions. 6.The light field imaging device of claim 4 wherein the zoom lens systemincludes a continuous zoom providing the plurality of zoom positions. 7.The light field imaging device of claim 4 wherein the light field sensormaintains the predetermined location relative to the focal point of theoptics for the plurality of the zoom positions of the zoom lens system.8. The light field imaging device of claim 4 further including amechanical system, coupled to the light field sensor and/or the optics,to maintain the light field sensor unit at the predetermined locationrelative to the focal point of the optics for the plurality of the zoompositions of the zoom lens system.
 9. The light field imaging device ofclaim 8 further including: a spatial adjustment unit, coupled to thelight field sensor and/or the optics, to responsively move the lightfield sensor unit; and control circuitry, coupled to the spatialadjustment unit, to control the spatial adjustment unit to maintain thelight field sensor unit at the predetermined location relative to thefocal point of the optics.
 10. The light field imaging device of claim 9further including memory, coupled to the control circuitry, to storedata which is representative of a plurality of locations of the lightfield sensor corresponding to a plurality of associated zoom positions.11. The light field imaging device of claim 9 further including memory,coupled to the control circuitry, wherein the memory includes a databaseor lookup table which correlates a plurality of locations of the lightfield sensor according to a plurality of associated zoom positions. 12.The light field imaging device of claim 11 wherein the control circuitryaccesses the database or lookup table having data which isrepresentative of the location based on the zoom position of the zoomlens system and generates control signal which are applied to thespatial adjustment unit to responsively move the light field sensorunit.
 13. The light field imaging device of claim 8 wherein, inresponse, to outputting the first image, the user interface receives thesecond user input which is indicative of the second virtual focus depth.14. The light field imaging device of claim 1 further including amechanical system, coupled to the light field sensor and/or the optics,to maintain the light field sensor unit at the predetermined locationrelative to the focal point of the optics for a plurality of focuses ofthe optics.
 15. The light field imaging device of claim 14 wherein themechanical system is active.
 16. The light field imaging device of claim1 wherein the optics is configurable to include a plurality of zoompositions which provide a plurality of different focal lengths havingassociated focal points and the device further includes a mechanicalsystem, coupled to the light field sensor, to maintain the light fieldsensor unit at the same fixed, predetermined location relative to thefocal point of the optics for the plurality of focuses.
 17. The lightfield imaging device of claim 16 wherein the mechanical system is activeor passive.
 18. The light field imaging device of claim 1 wherein theuser interface includes a display to (i) receive the data which isrepresentative of the first image and (ii) output the first image. 19.The light field imaging device of claim 1 further including memory,coupled to the processing circuitry, to store the data which isrepresentative of the second image.
 20. The light field imaging deviceof claim 1 wherein the processing circuitry computes output imageshaving an array of output pixels, and wherein the light field sensorincludes (i) a microlens array having a plurality of microlenses and(ii) a sensor having an array of sensor pixels, and wherein thepredetermined distance is between(0.7*N_(usable)*W_(MLA)*F#_(optics)*m)/(W_(output)) and(3.0*N_(usable)*W_(MLA)*F#_(optics)*m)/(W_(output)), where: N_(usable)is equal to the number of pixels containing usable directionalinformation across a microlens disk image that is formed, through eachmicrolens, on the sensor, or are located under a microlens, W_(MLA) isrepresentative of a number of microlenses across a width of themicrolens array, W_(output) is representative of a number of pixelsacross a width of the array of output pixels, m is equal to a distancebetween the centers of two neighboring microlenses in the microlensarray, and F#_(optics) is equal to an F-number of the optics.
 21. Thelight field imaging device of claim 1 wherein the light field sensorincludes (i) a microlens array having a plurality of microlenses and(ii) a sensor having an array of pixels, and wherein the predetermineddistance is greater than (m*F#_(optics)), where: m is equal to thedistance between the centers of two neighboring microlenses in themicrolens array, and F#_(optics) is equal to an F-number of the optics.22. The light field imaging device of claim 1 wherein the light fieldsensor includes (i) a microlens array having a plurality of microlensesand (ii) a sensor having an array of pixels, and wherein thepredetermined distance is less than (N_(usable)*m*F#_(optics)), where:N_(usable) is equal to the number of pixels containing usabledirectional information across a microlens disk image that is formed,through each microlens, on the sensor, or are located under a microlens,m is equal to the distance between the centers of two neighboringmicrolenses in the microlens array, and F#_(optics) is equal to anF-number of the optics.
 23. A light field imaging device for acquiringlight field image data of a subject in a scene, wherein the subject isin a focal plane, the device comprising: optics, wherein the opticsincludes an optical path, an optical depth of field and an autofocusmechanism, wherein the autofocus selects between a plurality of focusesincluding a first focus which is related to a focal plane of thesubject; a light field sensor, located (i) in the optical path of theoptics to acquire light field image data and (ii) at a predeterminedlocation, relative to the optics, during acquisition of the light fielddata, wherein the predetermined location is at a substantially fixedseparation relative to the focal plane corresponding to the subject, andwherein the focal plane of the subject is outside of the optical depthof field of the optics with respect to the light field sensor; andprocessing circuitry to generate an output image data which isrepresentative of an image including the subject in the scene, theprocessing circuitry to: determine a virtual focus depth of the lightfield image data, wherein the virtual focus depth is different from theoptical focus depth of the light field image data, and automaticallygenerate data which is representative of the image of the scene usingthe light field image data and data which is representative of thevirtual focus depth, wherein the image includes a focus whichcorresponds to the virtual focus depth.
 24. The light field imagingdevice of claim 23 further including memory, coupled to the processingcircuitry, to store the data which is representative of the image of thescene.
 25. The light field imaging device of claim 23 further including:a user interface to receive a plurality of user inputs, including firstand second user inputs, wherein, in response to the first user input,the light field sensor acquires the light field image data of the scene;and wherein the processing circuitry outputs the data which isrepresentative of the image of the scene, and, after outputting the datawhich is representative of the image of the scene and in response to thesecond user input, determines a final virtual focus depth of the lightfield image data using data which is representative of the second userinput, wherein the second user input is indicative of the final virtualfocus depth, and generates data which is representative of the finalimage of the scene using the light field image data wherein the finalimage includes a focus which corresponds to the final virtual focusdepth.
 26. The light field imaging device of claim 23 further includingcontrol circuitry and wherein: the autofocus mechanism includes anautofocus image sensor to acquire focus data, the control circuitrydetermines focus metrics using the focus data, and the optics includes asecond optics to provide at least a portion of the light in the opticalpath onto the autofocus image sensor.
 27. A light field imaging devicefor acquiring light field image data of a subject in a scene, whereinthe subject is in a focal plane, the device comprising: optics, whereinthe optics includes an optical path, an optical depth of field and anautofocus mechanism, wherein the autofocus selects between a pluralityof focuses including a first focus which is related to a focal plane ofthe subject; a light field sensor, located (i) in the optical path ofthe optics to acquire light field image data and (ii) at a predeterminedseparation, relative to the optics, during acquisition of the lightfield data, wherein the focal plane of the subject is outside of theoptical depth of field of the optics with respect to the light fieldsensor; control circuitry to adjust the separation between the lightfield sensor and the optics between a plurality of discrete, fixed-focusseparations, wherein each discrete separation provides a refocusingrange of the light field image data; and processing circuitry togenerate an output image data which is representative of an imageincluding the subject in the scene, the processing circuitry to:determine a virtual focus depth of the light field image data, whereinthe virtual focus depth is different from the optical focus depth of thelight field image data, and automatically generate data which isrepresentative of the image of the scene using the light field imagedata and data which is representative of the virtual focus depth,wherein the image includes a focus which corresponds to the virtualfocus depth.
 28. The light field imaging device of claim 27 wherein thediscrete, fixed-focus separations overlap in refocusing range of thelight field image data.
 29. The light field imaging device of claim 27wherein the control circuitry selects a discrete, fixed-focus separationbased on which of the discrete, fixed-focus separations provides a rangeof refocusing which is the furthest towards optical infinity and thein-focus plane of the subject is within the range of refocusing of theselected discrete, fixed-focus separation.
 30. The light field imagingdevice of claim 29, wherein: the autofocus mechanism includes anautofocus image sensor to acquire focus data, wherein the controlcircuitry determines focus metrics using the focus data, and the opticsincludes a second optics to divert at least a portion of the light inthe optical path onto the autofocus image sensor.
 31. The light fieldimaging device of claim 27 further including: a user interface toreceive a plurality of user inputs, including first and second userinputs, wherein, in response to the first user input, the light fieldsensor acquires the light field image data of the scene; and wherein theprocessing circuitry outputs the data which is representative of theimage of the scene, and, after outputting the data which isrepresentative of the image of the scene and in response to the seconduser input, determines a final virtual focus depth of the light fieldimage data using data which is representative of the second user input,wherein the second user input is indicative of the final virtual focusdepth, and generates data which is representative of the final image ofthe scene using the light field image data wherein the final imageincludes a focus which corresponds to the final virtual focus depth. 32.The light field imaging device of claim 27 further including controlcircuitry and wherein: the autofocus mechanism includes an autofocusimage sensor to acquire focus data, the control circuitry determinesfocus metrics using the focus data, and the optics includes a secondoptics to provide at least a portion of the light in the optical pathonto the autofocus image sensor.
 33. A light field imaging device foracquiring light field image data of a scene, wherein the light fieldimage device includes a maximum output image resolution, the devicecomprising: optics, wherein the optics includes an optical path and afocal point; a light field sensor, located (i) in the optical path ofthe optics to acquire light field image data and (ii) at a substantiallyfixed location relative to the focal point of the optics, wherein thesubstantially fixed location: (a) is substantially independent of thescene and (b) creates an optical depth of field of the optics withrespect to the light field sensor that extends to a depth that is closerthan optical infinity; and processing circuitry to generate and outputimage data which is representative of an output image of the scene, theprocessing circuitry to: determine a virtual focus depth of the lightfield image data, wherein the virtual focus depth is different from theoptical focus depth of the light field image data, and automaticallygenerate data which is representative of the output image of the sceneusing the light field image data and data which is representative of thevirtual focus depth, wherein the output image includes a focus whichcorresponds to the virtual focus depth; and wherein the light fieldsensor, which is located at the substantially fixed location relative tothe focal point of the optics, acquires light field image data whichcorresponds to an output image of the scene which includes a virtualfocus of optical infinity and a resolution in lines per picture height(LPH) that is at least 0.5 times the maximum output image resolution ofthe light field imaging device.
 34. The light field imaging device ofclaim 33 wherein the maximum output image resolution of the device is anumber of rows of pixels in an image output by the device.
 35. The lightfield imaging device of claim 33 wherein maximum output image resolutionof the device is the maximum resolution in LPH of any image output bythe device virtually refocused to any subject depth.
 36. The light fieldimaging device of claim 33 wherein the light field sensor, which islocated at the substantially fixed location relative to the focal pointof the optics, acquires light field image data which corresponds to anoutput image of the scene which includes a focus of optical infinity anda resolution in lines per picture height (LPH) that is less than 0.8times the maximum output image resolution of the light field imagingdevice.